Developing an ITS Technology Web Portal for Transit System Leaders (2016)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

May 2016 77 Test cases, and the associated test procedures, are divided by the overall portal function (e.g. Manage Access, Manage Storage, Manage Content, Manage Discovery, Administer the System). Detailed test procedures can be found in Appendix C. Chapter 4 :  Findings and Applications (Phase 2)   This section discusses the second phase of research. This phase involved building and testing a prototype of the portal that was specified in Phase 1. In addition, several supporting documents were developed to describe the operation and maintenance of the portal. These supporting documents included:  Implementation Plan  One-Year Marketing Plan  Sustainability Plan Portal Development  As part of the second phase of this research effort, the portal prototype was developed using a third party hosting service. The portal prototype was built by creating custom lists and workflows in Microsoft SharePoint 2010 as defined by the High-Level Design. The design for the portal was laid out by the work performed in Phase 1 of this research effort. The key documents developed in phase 1 all play a role in development of the portal. The concept of operations defines the user needs, and defines the overall use of the portal that must be met for the prototype to be successful. The functional requirements describe exactly what it is that the system must do. The technology alternatives analysis describes the technical requirements for the portal. Finally, the high- level design defines the lists and architecture of the system that must be implemented. In any system designed using the systems engineering process, the functional requirements are the critical document to describe the system. This is because the functional requirements provide the explicit definition of what the system must do to meet the ultimate needs of the users. All of the documents, including the functional requirements, were written with the intent of the portal eventually being hosted as part of MyAPTA. However, the goal of the research effort is to create a prototype of the portal, which can be used to validate the user needs of the Concept of Operations. Therefore, for the prototype, not all requirements were implemented. The requirements are prioritized, and it is anticipated that all high and medium priority requirements will eventually be implemented as part of the portal after the conclusion of this effort. Table 4-1 below shows the full set of functional requirements defined as part of Phase 1 and whether or not the requirements was included in the prototype are also shown.

May 2016 78 Table 4-1. Requirements Implemented in Prototype Reqmt ID Requirement Title Implemented in Prototype 1.1 Manage Access 1.1.1 Manage Portal Login Y* 1.1.2 Manage Roles Y 1.1.2.1 Assign User Role Y 1.1.2.2 Assign User to Multiple Roles N 1.1.2.3 Manage Role Privileges Y 1.1.2.3.1 Manage System Administrator Privileges Y 1.1.2.3.2 Manage Content Administrator Privileges Y 1.1.2.3.3 Manage Review Committee Member Privileges Y 1.1.2.3.4 Manage Content Contributor Privileges Y 1.1.2.3.5 Manage Content Reader Privileges N 1.1.3 Automate Role Grant Y 1.1.4 Manage Content Access Y 1.1.5 Manage User Access N 1.1.5.1 Manage Reviewer List N 1.1.5.2 Manage Committee Member List N 1.2 Manage Storage 1.2.1 Populate List Y 1.2.2 Validate List Entry Y 1.2.3 Manage Provisional Content Y 1.2.3.1 Restrict Access to Provisional Content Y 1.2.4 Manage Content Catalog Y 1.3 Manage Content 1.3.1 Manage Content Status Y 1.3.1.1 Designate Status Y 1.3.2 Manage Status Change Y 1.3.2.1 Document Status Change Y 1.3.2.2 Publish Status Change N 1.3.2.3 Manage Content Approval Y 1.3.3 Manage Content Registration Y 1.3.3.1 Manage Registration Process Y 1.3.3.2 Validate and Store Content Y 1.3.3.3 Manage Content Upload Y 1.3.3.4 Manage Content Agreement N 1.3.3.4.1 View Terms and Conditions N 1.3.3.4.2 Print Terms and Conditions N 1.3.3.4.3 Reject Terms and Conditions N 1.3.3.5 Submit Content for Review Y

May 2016 79 Reqmt ID Requirement Title Implemented in Prototype 1.3.3.6 Manage Incomplete Content Submission Y 1.3.3.7 Manage Revised Content N 1.3.4 Administer Content Review Y 1.3.4.1 Manage Pending Content Y 1.3.4.2 Manage Content Review Assignment Y 1.3.4.3 Review Content by Selection Criteria N 1.3.4.4 Manage Content Committee Review Y 1.3.4.4.1 Edit Review Agenda Y 1.3.4.5 Provisional Acceptance Alert N 1.3.5 Manage Content Quality Review Y 1.3.5.1 Reviewer Summary Page Y 1.3.5.2 View Assigned Content Y 1.3.5.3 Manage Review Scoring Y 1.3.5.4 Manage Incomplete Scoring Y 1.4 Manage Discovery 1.4.1 Manage Content Availability Y 1.4.2 Manage Free Text Search Y 1.4.3 Manage Keyword Search Y 1.4.4 Manage Most Viewed Search N 1.4.5 Manage New Content Search N 1.4.6 Manage Content Catalog Search Y 1.4.6.1 Sort Catalog Content Y 1.4.7 View Multimedia Formats N 1.4.8 Print Multimedia Formats N 1.5 Administer the System Y 1.5.1 Create Lists Y 1.5.1.1 Create Topic Type List Y 1.5.1.2 Create Keyword Lists Y 1.5.1.3 Create Content List Y 1.5.1.4 Create a Reviewer List N 1.5.1.5 Create a Review Scoring List Y 1.5.1.6 Create Supplementary Resources List N 1.5.1.7 Create Association between Lists Y 1.5.1.8 Create a Lookup List Y 1.5.2 Update List Format Y (*) denotes that final portal login will be exclusively managed by MyAPTA, however is separate for the purposes of the prototype test. The portal environment was designed to provide an environment where the logical operations of the portal could be tested by users. It was not designed to have the final

May 2016 80 appearance that it will have once installed. It is possible that the final implementation will look and act differently from the prototype created as part of this research effort. Portal Design  The portal was designed using Microsoft SharePoint 2010. The structure of the portal can be found in Figure 4-1. Figure 4-1. Portal Prototype Upon entering the portal, users are presented with the main home screen, which provides access to the content library. During the development of the portal prototype the team considered several approaches to the detailed implementation of the portal and settled on a simple structure, much like the popular internet search engines. This structure, shown in Figure 4-2 has the material on the portal organized by a type parameter, much like a library is organized by subject. The initial set of types defined for the portal were:  Application- content primarily relates to a specific transit application (e.g. Transit Signal Priority)  Benefit- content primarily relates to a discussion of benefits of some transit system  Information- content primarily relates to transit data or information  Solution- content primarily relates to a particular transit solution (e.g. Transit Enterprise Architecture)  Standard – content primarily relates to an ITS standard  Technology- content primarily relates to a specific technology applied to transit (e.g. automatic vehicle location)

May 2016 81 Figure 4-2 Portal Library Users can search for content by searching, keyword, or by browsing the library, which is organized by content types. Users who wish submit content can do so on the portal. After uploading the content, the user is provided with a form, shown in Figure 4-3, so that the content can be appropriately described and classified. As part of the content submittal, the submitter provides information about the content including:  Title  Keywords  Description  Date Published  Content Author  Topic Type

May 2016 82 Figure 4-3 Content Submission Form Content that has been submitted is stored in the “Landing Zone”, shown in Figure 4-4, which is an area of the portal for content which has not yet been designated for review.

May 2016 83 Figure 4-4 Landing Zone Content in the landing zone is accessed by the content administrator on the portal. The content administrator has the option to perform a security check for viruses or malware at this point. After this point, content can be placed in the review area, where it is assigned for review. The content administrator has the ability to assign the content for review. After content is reviewed, the content administrator can designate the content to be discussed by the full review committee for approval.

May 2016 84 Figure 4-5 Content Review Committee Assignment Once content has been reviewed by the review committee, it is published to the library and made available for all approved users to view. Verification Testing  After the portal was developed, the research team performed the verification tests defined in the Test Plan and Procedures document defined in Phase 1. Any deficiencies found in the prototype were rectified. Once the verification tests were deemed completed, the validation tests (beta tests) were performed. Validation Testing (Beta Tests)  In order to validate that user needs for the portal were met, validation tests were performed. A group of seventeen representative users of the portal was assembled. These beta testers were assigned to one of four roles:  System Administrator (2 testers)  Content Administrator (3 testers)  Content Reviewer (9 testers)  Content Contributor (3 testers)

May 2016 85 Note that the system administrator role was assigned to APTA staff, as they will be the system administrators when the portal is implemented in its final location. Also note that the content reader role was used as part of the beta test. This was because any user (reader) of the portal can also be a content contributor. The portal was available for user testing for a two month period. Overall, six sets of comments were received. The remainder of testers provided no comments. The comments received indicated a general satisfaction with portal operations. Most issues were reported were technical in nature, and will be accounted for during the transition to the final portal location. A summary of comments is in the table below. Table 4-2. Comments Received from Beta Testing Comment Response Check in screen disappeared when I closed a pop- up that was prompting me to consider Add-ons for my browser, IE 11 (which worked, by the way). Portal hosting site had some access issues. Should not be an issue when integrated into MyAPTA. I was able to login and access the portal as a reviewer smoothly. The response time seemed to be longer than expected (20-30 seconds?) from a user perspective. Portal is currently hosted in a temporary location. Speeds will be different when integrated into MyAPTA. Tester “wasn’t able to load the document as a contributor intuitively”. Got an error message. We have loaded documents without issues, so likely an issue of upload to temporary site. There is a 255 character limit on the Content Description field. Agree this should be changed to allow longer Content Description. Content submitters may need to revise their work after they see the results. Need to clarify how to go back and edit previously written entries Agree that some indication of how to edit entries should be added. The “keywords” link on the landing page doesn’t work This is likely an error that would be fixed. The Content Administrator, Content Contributor, and Content Reviewer all appear to have the same permission level – contribute. That doesn’t seem correct. What should they be? It would be ideal if we could use OOB security permission levels. Permission levels were limited due to simplified setup of beta site. Production version would have separate security permission levels. Description of roles and responsibilities inadequate These descriptions were for the Beta Test only. Roles should be clarified on the final implementation. The majority of people accessing the portal in the long run will be a contributor only. System worked as expected - no issues, but it did take me a few minutes to figure out what the desired action was in my packet. A summary of what the intent of the action and expected result would have been helpful. Beta Test Operating Procedures were meant to provide step-by-step instructions, but in some cases weren’t as detailed as they could have been. Based on the results of validation test, the portal prototype is considered to meet the user needs described in the Concept of Operations developed in Phase 1 of this research effort.

May 2016 86 Operations and Maintenance Documents  Three documents were developed to describe how the portal should be implemented following the conclusion of this research, how the portal should be marketed to ensure its success, and sustainability issues associated with the portal, which provided considerations for maintenance of the portal. Portal Implementation Plan   As part of this research project, a test version of the Portal was developed and beta tested. Following the completion of this project that test version will need to be transitioned to MyAPTA for its final implementation. The purpose of this Implementation Plan is to identify the resources and issues relating to this transition. Portal Operations  The Portal will become a part of the current MyAPTA environment that is owned and operated by APTA. The Portal and its contents will be under the control of the Technical Portal Subcommittee of the APTA Research and Technology (R&T) Committee. In its operation, the Portal will have the following types of users:  Content Contributor: The content contributor can be any APTA member and is responsible uploading content to the portal. Note the original Concept of Operations included a Content Reader class of user, who can search and view content on the sites. As the Test site was developed, it was determined that Reader and Contributor were really the same class of user and so they were combined into a single class on the Test Portal. Any APTA member can view the Portal and can contribute to the portal. The recommendation is that this be continued in the final implementation.  Content Reviewer: The content reviewer is responsible for reviewing content that is submitted to the portal. Theirs is the key role of vetting material submitted to the portal so that only material meeting Subcommittee criteria of relevance and quality is included in the portal. The Content Reviews also can perform the functions of Content Contributor (or Reader).  Content Administrator: The content administrator role is responsible for guiding content through the review/approval process. This includes assigning content for review and managing the final approval process including publication of the material to the final site. The Content Administrator can also act as a Content Reviewer or Content Contributor if desired.  System Administrator: The system administrator role is responsible for the technical maintenance of the portal. This role involves a detailed knowledge of SharePoint, and is intended for APTA staff who will be maintaining the final version of the portal.

May 2016 87 The policies and procedures to be used for the operation of the Tech Portal are defined by a Tech Portal Governance Policy. A draft version of the Governance Policy developed as part of the research project is contained in Appendix A. Portal Rollout Plan    In order to implement the Technology Portal on the MyAPTA site, the following Rollout Plan steps are suggested:  Scoping the transition- APTA/Team discussion  Transitioning from Beta Test Site to Operational Site  Testing of Operational Site  Initial Operations  Operations through end of Year 1 The discussion below elaborates on this suggested set of steps. Scoping the Transition The team suggests a meeting with APTA to scope the issues that will be associated with the transition. The beta test yielded a few issues that must be addressed prior to implementation in the APTA environment as described below. Transitioning from Beta Site to Operational Site The Portal Beta Test Site has been designed without the use of custom code, rather it is based on customization features that exist in the capabilities of SharePoint 2010. The result of this design is captured in the Portal Beta Test screenshots that are contained in Chapter 4 of this document. These, augmented with work flow diagrams developed to create the Beta Test Portal will be the key outputs used to transfer design of the Portal Beta Test Site to APTA for creation of the Operational Site. The following are key issues that will need to be resolved in making this transition:  SharePoint FAST Search Capability: The Portal Beta Test Site uses the FAST capability of SharePoint 2010. This allows quicker searching of the documents on the site. It was our understanding that the APTA SharePoint site had this capability. Once the Portal Beta Test Site had been developed, we learned that, while the APTA system could implement this, they do not currently have it implemented. Some revision of the site design will be needed to transition from the Beta Test Site with FAST search to the APTA site without it.  Permissions by User Class: The Portal Beta Test Site did not have the capability to define unique class permissions for the three areas of the site: Landing Zone Library, Review Zone Library, and FAST Search Library. The APTA site will need to be configured to allow different classes of uses to have different permissions in these three Library areas.  Setup of the metadata or keywords may require turning that feature on within the APTA instance of SharePoint. There should be agreement on what level of detail is available to what level (roles and permissions) of user at that point.

May 2016 88 The labor cost to transition from the Beta Site to the Operational Site is not a part of the current research project. The amount of labor required to work with ATPA IT Staff to effect the transition can be more precisely estimated one the developer of the Beta Test Site meets with APTA to better scope the transition. Testing of Operational Site Once the initial Operational Site has been developed, it should be tested prior to commencing operations. Testing of the site could be done in two steps. First the set of test procedures (or an appropriate subset of them) shown in Chapter 3 should be performed to verify the basic functionality of the site. As part of this testing a set of documents (and other content as available) should be loaded onto the site. The documents used to seed the Portal Beta Test Site could be used for this purpose. Once the site verification is complete, then a short beta test performed by members of the Tech Portal Subcommittee should be held to ensure that the initial operations flow of the site is working. Initial Operations Prior to beginning initial operations, it is recommended that the Tech Portal Subcommittee meet at least once to decide on a few site specifics prior to beginning the Operations:  Agreement on the list of Topics (essentially the “shelves” around which the Library is organized. The Beta Test Site had 5 topics, partly driven by the seed material used.  Solicit additional material from subcommittee members so that when first going operational there is a critical mass of material present to interest the general user. Once the initial content has been loaded the recommendation is that the Marketing Plan below be consulted on the approach to announcing the availability of the Portal. Operations Through End of Year 1 During the first year of operations, it is recommended that the Tech Portal Subcommittee meet at least monthly to review the status of the Portal. One Year Marketing Plan  Developing this important tool will build knowledge and collaboration in the ITS area. Promoting the APTA ITS Portal will be valuable in two ways. First, the more participation there is - the more robust and useful the site will be. Second, promotion reinforces APTA’s leadership position in the ITS area. The goal of the marketing plan is twofold:  Increase awareness of the APTA ITS Portal

May 2016 89  Encourage participation in the APTA ITS Portal This preliminary marketing plan focuses on attracting users to the site to meet both marketing goals. Marketing Strategy  Target Audience  The primary target audience for the APTA ITS Portal consists of current and prospective transit ITS middle and senior managers, practitioners, executives, and students. The secondary target audience is potential members who will see the site as an incentive to join APTA. APTA may also want to consider whether it is viable and worthwhile to expand the portal concept with two levels of access. The first level would be open to all transit professionals and researchers to promote the awareness and encourage submittal of ITS- related articles. It would have limited functionality that would still provide additional value for APTA members, who could access a “members only” version of the site. Message Development  Selecting and focusing on a key message point will help APTA cut through the immense clutter everyone faces and help create a clear benefit message and compelling reason to try the APTA ITS Portal. The key attributes to emphasize in communications are:  This is an innovative concept from a trusted source  This is a collaborative site where you can gain knowledge about technology in the industry  This is where to go to get the best information  This is an opportunity to find up to date information for members who can rarely attend conferences or trade shows  Submissions are encouraged on a continuing basis Message Distribution  As this will be a member-only portal, the message should be distributed through APTA communications, including general membership and committees. We recommend engaging agency board members, CEO’s, and VP’s, many of whom are highly involved with APTA already, to become champions for the ITS Tech Portal and promote the site through their organization and internal distribution channels. Measurement  The first year program will be used to establish a benchmark for future promotional efforts. We recommend that we track activity that follows from outreach efforts to gain knowledge about which techniques are working the best. Throughout the year, visits to the website will be tracked and compared with the timing of the launch of each promotional activity to determine which effort yielded the greatest

May 2016 90 results. At the end of the measurement period, a summary of the activity will be presented and used to develop future marketing plans. Creative Strategy  Brand Management  We recommend development of a distinct branding identity for the APTA ITS Portal consisting of a name and visual elements. This will help to create a memorable, easily recalled mark that can be incorporated into communication materials. It will also position the Portal in a similar vein to an application, something that people can relate to as a stand-alone tool. This will set the stage for APTA to further develop the portal as a stand- alone app if that is desired in the future. Creative Execution  Messaging should position the ITS portal as a fresh, approachable tool for use by technical and non-technical audiences. As the most likely early adopters will be transit- oriented individuals who really care about leveraging technology to serve transit customers, we recommend a creative campaign that incorporates an educational element (to take the fear out of learning new technology) with edgy, cheeky, non-traditional marketing to reinforce that this is a new type of offering. Complementing this would be testimonials, such as endorsements, from highly regarded transportation C-level personnel and other substantiating information to support the business case for using the Portal. Media  To reach the primary target market, we recommend using a mix of traditional and social media with the intention of building a community through social marketing and word of mouth promotion. APTA could begin with its own membership list, sending blast emails as well as messaging to all committee chairs. Facebook and Twitter accounts should be set up and used to foster a conversation around the portal. Resources will have to be dedicated to manage and monitor these social media sites. Using viral social marketing techniques to build “buzz” will draw viewers to the portal. APTA could also consider the use of contests or awards (e.g., “most innovative technology”) to build value for participating in the site. Members, especially agency members, could be encouraged to share information about the site with their staff through their internal communication mechanisms. APTA should also consider booth displays at APTA events. A branded giveaway item, such as a flash drive, would help reinforce the portal identity and provide a way to remember the site. In addition, APTA should incorporate the ITS Technology Web Portal into its benefit package for marketing to potential members.

May 2016 91 Portal Sustainability Plan  A technology portal is long-term ongoing investments that will grow and change as both the content and underlying technology evolve. The purpose of this portal sustainability plan is to examine issues relating to long-term usage of the portal. Consideration of these issues before the portal goes live will help to ensure both short- and long-term value to the users. Two key goals of portal sustainability are:  Ensuring that the service provided by the portal (content and functionality) provides sufficient utility to all categories of users  Ensuring that the software/hardware is sufficient to handle portal functionality in the near term and a projected 5-year future term To address these goals, this Plan will consider three major issues relating to portal sustainability:  Portal Operations and Maintenance Resource Requirements. Portal Maintenance Resource Requirements describe the requirements for the ongoing Operations and Maintenance of the Portal.  Portal Upgrade Planning. Portal Upgrade Planning describes changes that could be made to the portal to enhance portal operations.  Ongoing Portal Evaluation. Ongoing Portal Evaluation describes the process where operations are assessed to make decisions about the future of the portal. Portal Maintenance Resource Requirements  The Portal Maintenance Resource Requirements can be broken into two areas:  Human Resource Requirements  System Resource Requirements Another possibility exists to transition some of the human or system resources to a cloud- based “as-a-service” paradigm. This paradigm shift should be considered as part of a long-term sustainment of the portal. The dynamics of shifting to this can effect change on both the human and system resources. Human Resource Requirements  Human resource requirements are those that require personnel to perform activities required to maintain the portal. Personnel may be paid APTA staff or volunteers (Reviewers and Review Committee Members). The Transit Technology Portal has been designed to operate within existing SharePoint capabilities of APTA, not as a separate system. Nonetheless there will be some O&M requirements to the portal itself. In addition, existing content of the portal must be managed and new content must be reviewed and approved for inclusion in the portal. This content review and management will also require ongoing resource commitments. These resource requirements can be summarized by three key roles associated with the Portal:  System Administrator  Content Administrator  Content Reviewer

May 2016 92 The requirements for each of these key roles are listed below:  System Administration. A System Administrator (who will likely be a member of APTA Staff) will perform routine tasks to ensure that the portal is operational. This includes installing software updates provided by vendors, performing security operations on the portal (e.g. scanning content for viruses), responding to technical support inquiries by portal users, and resolving technical issues that occur. The cost of system administration is the labor cost of staff members required to perform the activities, and the required level of effort is variable based upon the activity of the portal and how well integrated the portal is to the APTA website. This is a typical role where the labor can be greatly reduced through cloud-based services and should be reviewed in the long-term sustainability of the portal. The additional resource requirement (above and beyond the existing SharePoint system that APTA operates) will likely be small – a few hours per week.  Content Administration. In order for the portal to be an effective resource to its users, an ongoing content management and review process is required. This includes reviewing content submitted by users, approving content, publishing content, and ensuring the content is up to date. This work is led by a Content Administrator and supported by content reviewers and members of the review committee. The role of Content Administrator could be performed by APTA staff, but will likely be done by an APTA member as an unpaid volunteer. The Content Administrator may be the Chair of the Tech Portal Subcommittee of APTA’s Research and Technology Committee or someone to whom the Chair delegates the responsibility. Early in the deployment process the amount of effort required for this role will be a function of how much new content is submitted to the portal each month. Once the portal has been in operation for more than a year, then a portion of the effort will be to review existing material to consider whether it needs to be removed, revised, or left unchanged. Another factor affecting the resource requirements will be how often the Review Committee meets to consider new submittals to the portal. The most logical meeting frequency is monthly, although in the early months of the portal meeting bimonthly may be worthwhile if there are many submittals to consider. The content administrator has two primary roles – to lead the submittal review effort and to manage the status and location of portal material. It is estimated that leading the review effort (which is primarily administrative in nature) should take no more than 4 hours per review period (e.g. either bimonthly or monthly). The additional effort to administer the content itself should amount to no more than a couple of hours per week.  Content Reviewers: Content Reviewers are needed to review submittals to approve their inclusion in the portal. The Content Reviewers will likely be drawn from members of the Tech Portal Subcommittee of APTA’s Research and Technology Committee, although they could include other transit professionals

May 2016 93 invited by the Subcommittee to participate as reviewers. These Reviewers provide technical or quality review of submittals. The amount of effort required will depend on two factors – how many submittals need to be reviewed by an individual reviewer (will be affected by number of submittals and number of reviewers) and the amount of effort required to review each submittal. Once the portal has transitioned to APTA, Subcommittee members will need to decide how comprehensive a review they will want on the submittals. The review could be a quick look for quality or appropriateness, or the Subcommittee might ask for a more complete review of each submittal. In either case the effort will have to be supplied by the volunteer group of reviewers. System Resource Requirements  System Resource requirements account for the ongoing technology needs of the portal. The exact long-term System Resource Requirements for maintenance of the portal are unknown and will vary based on portal utilization. There are a number of system resources that can be greatly reduced through cloud-based services and should be reviewed in the long-term sustainability of the portal. They range from cloud-based backups (of portal content) to complete hosting through PaaS (Platform as a Service).Throughout the lifecycle of the portal, the following requirements are possible:  Storage Media Upgrades. In the event that the portal experiences extremely high utilization, it may be necessary to increase the available storage space. The probability of this routine upgrade increases if the amount of content uploaded to the portal is greater than expected or the average size of content increases. The cost of such upgrade would be that of new media plus the labor to make the changes. Another plausible future state would be to leverage cloud-based storage that expands or contracts based on needs and record storage longevity  Routine Software Updates. It is possible that an update may be released for the SharePoint or FAST software items that run the portal to fix an issue in the existing software. This is not the same as a full upgrade of the version of SharePoint or FAST. The cost would likely only be the labor cost to perform such upgrades, and would probably not be unique to the portal but would represent a cost for APTA’s overall SharePoint system. Another possible long- term objective would be to leverage cloud-based services to host the site as this would obviate the need for any software updates (provided by vendor through service-level agreement (SLA))  Hardware Repair/Replacement. A possible, but unlikely, cost is that the hardware that the portal operates off of could have hardware problems requiring repair or replacement. However, since the hardware supporting the portal is just part of the hardware supporting the overall APTA website, the cost associated with repair or replacement is likely not directly associated with the portal. If hardware becomes an issue, at this juncture it would certainly be worth exploring hosting the system(s) through PaaS (Platform as a Service).

May 2016 94 Portal Upgrade Planning  For the portal to remain a viable asset for APTA, planning for upgrades needs to be considered as part of the overall sustainability of the portal. Upgrades may be needed to hardware, software, or functionality/content in order to maintain ongoing interest in the portal. Update Schedule  At this time, a specific upgrade schedule for the portal is not warranted because system utilization is not yet known. While both hardware and software upgrades may be necessary at a future date, these should be driven by the portal utilization which is not known at the present time. While an exact timeframe for upgrades is not known and is dependent on system performance, it would be most efficient to perform such upgrades concurrently with any other planned upgrades to the APTA website. This takes advantage of economies of scope, as it is much more time- and cost-effective to perform work at the same time. This should help minimize disruptions to the portal, and other APTA systems. Further, if an upgrade to Microsoft SharePoint is performed as part of an APTA website upgrade, the portal will be directly affected, as it is specifically built to work on the version of SharePoint currently used by APTA. Hardware Upgrades  The initial use of the portal will be supported by the existing hardware configuration of APTA systems. This configuration is likely to support portal activity for the near term depending on the level of usage, which should be monitored as part of performance evaluation. If the portal usage increases significantly over time then the hardware configuration may need to be upgraded but, as discussed above, any hardware upgrades should be done as part of larger APTA upgrade efforts or planned in a cloud-based roadmap. Software Upgrades – Impact of Upgrading to a New Version of SharePoint  The current portal was built on SharePoint 2010, which is the current version of SharePoint used by APTA. Software upgrades will not be needed in the next couple of years, as that version of SharePoint should be supported for at least several more years. If APTA should decide to upgrade to the next version of SharePoint, which is SharePoint 2016 the features used for the portal would have to be tested to ensure compatibility across the entire APTA site. Content or Functionality Upgrade  The portal will likely have limited content at its first introduction. This effort will provide some seeding of content, and the current APTA site has content in various locations that can be consolidated and organized onto the portal. One key to

May 2016 95 sustainability for the portal will be for new content to be added in the first year of operations. The Portal One-Year Marketing Plan, previously developed and submitted, provides a plan for increasing awareness of and participation in the portal. The two most likely sources of new content are, first, from the members of the Technology and Research Committee and, secondly, from general APTA portal users. During the first year of portal operations, the committee should actively seek additional comments from its members and a recommendation is that discussion of the portal and its status (including evaluation measures mentioned below) should be an active topic of discussion at committee meetings. Regarding functionality, the initial ITS Portal design has been developed with a base set of functionality as described in the System Requirements documentation. During the detailed design and development of the portal, the development team ran into a number of areas where design choices had to be made within the basic set of requirements. Some of these design choices were presented to the panel at a web conference last summer and our initial design choices were confirmed. Since then we have continued to identify and resolve issues associated with the site design. As the capabilities of the site are utilized over the first year of operations, it is likely that the subcommittee responsible for maintaining the portal will decide that changes should be made to the Standard Operating Procedures defined for the prototype or to the portal design itself. The prototype portal is built around existing SharePoint capabilities; should simple changes be required within the structures of SharePoint these will be well within the capabilities of APTA’s staff that maintains the organization’s overall SharePoint site. A recommendation is that the subcommittee charged with maintaining the portal meet quarterly or semiannually to discuss any changes or upgrades that should be made to improve the usability of the portal by content readers, content submitters, or content reviewers. Ongoing Portal Evaluation  In order for the portal to be successful, it is important that it be assessed on a routine basis. This helps determine how well the portal is operating and identifies any issues that must be resolved. Ongoing evaluation will also give insight into how the portal is being used and allow it to be improved or updated. Evaluation of the portal should be both quantitative and qualitative. Quantitative evaluation will consist of data that is collected about portal usage, which can be compared against benchmarked performance measures. This provides a snapshot of how well the portal is operating. Qualitative evaluation, in the form of user feedback, can provide insight into user satisfaction with the site. It is important for evaluation to occur on a regular basis. By routinely looking at system performance, issues can be identified early and positive attributes of the portal can be capitalized upon. Evaluation Methodology   Quantitative evaluation for the portal can be performed by collecting raw usage data from the portal itself. Every time a user interacts with the portal, a new data point is created.

May 2016 96 Since all users must be logged in using their APTA accounts, demographic data is tied to each usage, which allows for an even more detailed analysis of who is using the portal. Further, general data about the portal, such as the total number of content items, can also be measured with ease. To perform this type of quantitative assessment would require some setup and continued monitoring of the portal. Any data collection activities should be performed in accordance with APTA policies, particularly as regards existing privacy policies. For example, it may be necessary to make usage data anonymous so that a specific user cannot be identified. It is recommended that each login to the portal be recorded. This should include demographic data. At a minimum, the user access level should be recorded. A timestamp should also be recorded. Additionally, a timestamp and identifier linking to a specific login should be recorded every time content is viewed. It should be recorded if the content is downloaded to the user’s computer or not. This recorded data should be paired with metadata about each content item, to provide insight into portal usage. From recorded data, combined with content metadata a variety of values can be derived. This includes, but is not limited to:  Total log-ins  Total content views  Total content downloads  Number of log-ins for each user  Number of content views for each content item  Number of content downloads for each content item  Total Storage Space Used In addition to automatically collected data, user feedback should be solicited on the portal, so that qualitative aspects can be monitored. Further, if a significant change in performance metrics occurs, the user feedback should be checked to see if it provides insight on the change. Performance Metrics  There are many possible performance metrics that can be used to evaluate portal operations. While a large quantity of data could be collected for the development of performance metrics, it is recommended that only a limited number of key performance indicators be utilized for general portal evaluation. This allows analysis to remain focused relative to the size of the portal. It is always possible for additional metrics to be observed. The following are some suggested key performance indicators for the portal. Note that target values are not given, since the utilization of the portal is not known. It is recommended that, after the portal is deployed for one year, benchmark values be determined which can be used to drive any targets. Performance indicators should look at future changes in these values relative to the benchmarked values.

May 2016 97  Total Weekly Visits. This metric analyzes the total visits to the portal each week. Patterns in weekly visits can be used to observe the effect of various portal events, such as a change in functionality. A rapid change in total visits can also highlight a problem with the portal that may have occurred. A weekly period is selected to limit noise created by different days of the week having different usage patterns.  Rolling Average Weekly Visits. This metric analyzes the average number of visits per week over the past year. This value is calculated by averaging the total visits each week for the past year, over the total number of visits for the past year. Since this is a rolling value, the past year is defined as beginning one year from when the value is calculated, as opposed to the calendar year. A weekly period is selected to limit noise created by different days of the week having different usage patterns. An average limits the effect of week to week variations in usage.  Monthly Unique Visitors. This metric analyzes the total number of unique visitors each month. This metric can give insight as to whether visits to the portal are coming from a broad spectrum of users or not.  Average Time Between Visits. This metric analyzes the average time between visits for all portal users who visit the portal two or more times. This gives insight into how frequently users are inclined to visit the portal, and check for new content.  Total Monthly Content Publications. This metric gives insight into the variety of new content available on the portal and can be correlated with other metrics to measure the effect of new content on readership.  Average Age of Published Content. This metric is an average of the length of time that all currently published content items have been published. This gives insight into whether or not content is fresh and likely to attract users to the portal. During the first year of operation, the values of each performance measure should be closely observed. In addition to monitoring system performance, these values can be used to measure the effectiveness of the portal marketing plan. At approximately one year, benchmark values should be determined, based on observed values throughout the first year. Future observed metrics should be compared against these benchmark values to determine performance. Target values can also be determined once benchmarked values are known. As the portal matures, it may be necessary to modify target values if there is sufficient reason to believe they no longer reflect the ideal operations of the portal.