Caseflow: Complete Product Score Card Metrics for OIT

[alisan16]

OIT has requested we complete the following "Product Score Card" metrics for Caseflow. Most of the metrics are for the time period July 1 - July 31, 2020, though the two about deployment releases is requested from January 31, 2019 - July 31, 2020. The metrics are due * August 3, 2020 *. There is a chance that these metrics will be requested on a regular basis, so a repeatable method for acquiring them is a nice to have for this ticket, though not required given the short time frame.

The product lead with the product management team will complete metrics 1 & 2. cc @jessalvesdesa
The engineering lead with the data functional lead will complete metrics 3 - 7

Acceptance Criteria

• [x] Edit the table below with the Caseflow Metric column filled in. Add comments to this ticket to track work
• [x] Alisa will email metrics to Fred, who will send it to OIT

| # | Status | Metric | VA Definition | Caseflow Metric |
|---|---|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|
| 1 | DONE | Average Lead Time | From time an item (story) is created on a product team backlog to the time it is resolved (1 month of data July 1, 2020 – July 31, 2020) Hint: If you want, send a report to me from your ALM tool which includes all user stories and defects “resolved” after May 23, 2020 20 (irrespective of when they were created or deployed to production). We can calculate the average lead time. The fields in the report/spreadsheet should include Id, Work Item Type, Summary, Status, Creation date, and Resolution date. | 80 days |
| 2 | DONE | Defect Lead Time | From time a defect is created on a product team backlog to the time it is resolved (1 month of data July 1, 2020 – July 31, 2020) Hint: If you send this data along with #1 as a report, we don’t need anything else. Otherwise, we need all Resolved Defects after May 23, 2020 (irrespective of when they were created or deployed to production). |22 days |
| 3 | DONE | Deployment Frequency/Release Cadence to Production | Average amount of time (days) between releases to Production (currently we will measure from January 31, 2020 to July 31, 2020) Hint: include patches. 3 and 5 go together. | 0.97 days |
| 4 | DONE | Mean Time to Restore | How long it takes to restore service for the primary application or service they work on when a service incident (e.g., unplanned outage, service impairment) occurs (current MTTR) Hint: Hypothesize, today if your system (the system that is under your control) went down; how long would it be for the system to come back up, on average. | We have not had any production incidents in July 2020 see full incident tracking here. For the most recent month with a system outage, May 2020, the mean time to restore was 81 minutes. Estimate 90 minutes |
| 5 | DONE | Change Fail Percentage | Total Production Releases by Month/number of releases that had to be backed out and/or rolled back to a previous build/version. (from January 31, 2020 to July 31, 2020 for releases from cadence, get number failed, rolled back) Hint: release data back 6 months; 3 and 5 go together. | 0 % |
| 6 | DONE | % Test Cases Automated | Number of automated test cases divided by (/) total number of test cases. | 99.8 % |
| 7 | DONE | Last Release Date | Last release date into IOC prod or production | 07/31/2020 |

[alisan16]

4 is done

We have not had any production incidents in July 2020 see full incident tracking here. For the most recent month with a system outage, May 2020, the mean time to restore was 81 minutes

[yoomlam]

For items 3 and 5, here is what I have so far (up to Nov 29, 2019 -- the limit of Jenkins logs). Results are in columns I through N.
Asked for prior logs from DevOps -- doesn't seem like we have other logs.
I'll update the spreadsheet on Friday to include up to July 31st.

[alisan16]

@yoomlam We have confirmation from OIT that they only need 6 months worth of data, from January 31, 2020 - July 31, 2020. No need to include anything prior to this year

[jimruggiero]

@alisan16 #1 and #2 are done. I've aggregated the raw data here.

Calculation parameters:
Average lead time

• Includes all issue types except bugs and epics, then
• Includes all issues either opened and/or closed in July 2020
• 1 day added to all open status issue lead times to account for data pull on 7/30 vs 7/31

Defect lead time

• Only includes bugs (no epics), then
• Includes all bugs either opened and/or closed in July 2020
• 1 day added to all open status bug lead times to account for data pull on 7/30 vs 7/31

Note: Two bugs are outliers that were left open for 330 and 372 days respectively for monitoring, which unreasonably skewed the average, so I eliminated them from the final calculation. You can see both numbers in the at the top of the google sheet with the raw data.

Let me know if you have any questions or need additional info.

[alisan16]

For #6

• There are 11 manual test cases set up for Caseflow-VBMS integrations (Dispatch, Intake, eFolder)
• Circle CI tests for Caseflow master as of 7/31/2020: 6650 total
  
  
  
  • rspec: 6515
  
  
  • js_tests: 135
  
  

So 6650 / 6666 = 99.8%

[yoomlam]

For items 3 and 5, updated spreadsheet through July 31st. Results are in column K.

[alisan16]

Thank you @yoomlam and @jimruggiero ! I'll get these to Stephanie and Fred.

[alisan16]

Email sent to Fred. @FredAllen6608 please send the metrics to Stephanie.

[alisan16]

Yes, this is becoming a monthly thing :(
Some of the Github issue metrics will be tracked by their team, but we are still being asked to provide the below on a monthly basis. I've filled in the numbers manually for August.

Number | Metric | VA Definition | Metric
-- | -- | -- | --
13 | Test Environment Matches Production | Purpose of question is to have a functionally equivalent of production, to development and test teams during testing. Does the test environment match the production environment in regard to platform, OS, applications, interfaces, data, and simulated networking? | Yes
14 | Overall Product Test Cases Count | Current product active (not archived) tests include: Functional, Smoke, Integration, System, Regression, and Unit test cases. | 7064 (6846 rspec tests + 218 jest tests)
15 | Performance and Load Test Cases Count | Number of product test cases that verify system performance requirements. | ##
16 | System and/or Application Production Monitoring is in place | Application monitoring capabilities (Performance, Status (Up, down, etc.), alerts, incident management integration). | Yes
21 | Number of Failed Releases | Number releases that had to backed out (revert to previous version) during August 1, 2020 – August 31, 2020. | 0