Caseflow: Defer the loading of user on_hold tasks

[lomky]

Description

Right now when a User navigates to their /queue, we grab all the open tasks assigned to the user plus all of their recently completed tasks. As time has gone on and users are doing more and more work in Caseflow, User on_hold numebrs are going up to a point that they are experiencing load times of nearly a minute, even though they only have a handful of tasks to immediately work.

This issue is to prioritize loading of active issues first and foremost, and defer to loading of on_hold and recently completedtasks until the user actively requests those tasks by clicking on their associated tab.

Acceptance criteria

• [ ] User queues load only the tasks needed for the default tab (Assigned/Assign/Review) when a user first loads /queue
• [ ] User on_hold tab still shows the correct number of on_hold tasks
• [ ] on_hold tasks are queried and displayed when the user selects that tab
• [ ] recently completed tasks are queried and displayed when the user selects that tab

Background/context/resources

Discussion thread

Technical notes

It's important to handle tasks that _were_ active but have since become on hold in a users queue, such as if they have assigned a case over to another team, and return those tasks so they are unloaded from the users active tab.
See the implementation of a similar need for Judge queues, though that ticket does not cover the second AC of loading on_hold on request.

[ajspotts]

_what is this chart?_

1 | 
2 | 
3 | ||||
5 | ||||||||
8 | 

Reminder: record load times before and after the improvement. (may be available in the DevTools Network console tab)

Why 3?

• Can borrow some "should load tasks" logic from pagination in org queues

Why 5?

• Imagine all the wrangling of tasks in state to determine if we should load tasks, front end requests, controller updates to allow tab or status to be specified, same for queue for role and the respective queues.
• Grabbing task counts for each tab may not already be done in queue config
• backend and front end changes, affects all three User queue types

[hschallhorn]

First option was to add pagination to user queues. Allows us to reuse a bunch of code we already have implemented. However. We cannot paginate judge and attorney queues as their queues also show legacy tasks, which are not persisted in our database and cannot be sorted/filtered. Judge queues are already mildly handled as we do not return on hold tasks to the front end. Let's see if any attorney queues are an issue.

Task.open.where(assigned_to_type: User.name).group(:assigned_to_id).order(:count).count
... 1605=>100, 2143=>106, 11350=>107, 1608=>121, 1607=>124, 3772=>170, 2062=>209, 10630=>211, 3482=>220, 14143=>226, 10627=>251, 2101=>259, 1379=>278, 1782=>285, 7798=>296, 1967=>302, 2141=>317, 2012=>341, 1829=>372, 6169=>424, 2014=>436, 546=>462, 1885=>483, 2227=>486}
power_user_ids = Task.open.where(assigned_to_type: User.name).group(:assigned_to_id).order(:count).count.keys.last(25)
power_user_ids.select { |id| User.find(id).attorney_in_vacols? && !User.find(id).judge_in_vacols? }
=> [2101]
# Looks like we only have one atty with over 100 tasks

power_atty = User.find(2101)
tasks = QueueForRole.new("attorney").create(user: power_atty).tasks
tasks.count
=> 293
Benchmark.measure { AmaAndLegacyTaskSerializer.create_and_preload_legacy_appeals(params: { user: power_atty, role: "attorney" }, tasks: tasks, ama_serializer: WorkQueue::TaskSerializer).call }
=> #<Benchmark::Tms:0x000000000b4436c0 @label="", @real=48.63590942599967, @cstime=0.0, @cutime=0.0, @stime=0.45306500000000005, @utime=11.936532, @total=12.389597>

[hschallhorn]

@araposo-tistatech The pagination of user (non attorney and judges) queues is merged and will be released after the weekend. This is behind a feature flag. Do we want to identify some users to test this out with before we enable it for all? Perhaps with the user from this thread would be a good start.

[araposo-tistatech]

@hschallhorn absolutely, I can't immediately identify the user in the thread (slack is not taking me directly to the thread for some reason), but I will request users from a few organizations we can test with.

[araposo-tistatech]

Confirming user request was sent.

[araposo-tistatech]

@hschallhorn below are the names of users I've received:

Michael Theriot
Chandra Tyler
Kristian Berhost
Robert Carter
Naheem Grant
Lazette Clanton
Ronald Bergeron
Andrew Kim
Shaliah Benton

[hschallhorn]

user_names = ["Michael Theriot", "Chandra Tyler", "Kristian Berhost", "Robert Carter", "Naheem Grant", "Lazette Clanton", "Ronald Bergeron", "Andrew Kim", "Shaliah Benton"]
users = User.where(full_name: user_names.map(&:upcase))
counts = Task.open.where(assigned_to: users).group(:assigned_to_id).order(:count).count
=> {11140=>29, 14385=>106, 14143=>241, 3482=>299, 6169=>415, 2014=>437}
counts.map { |k, v| User.find(k).full_name }
=> ["NAHEEM GRANT", "KRISTIAN BERHOST", "ANDREW KIM", "SHALIAH BENTON", "ROBERT CARTER", "RONALD BERGERON", "LAZETTE CLANTON"]

@araposo-tistatech Looks like our best bets are (in order of the most tasks)

1. LAZETTE CLANTON
2. RONALD BERGERON
3. ROBERT CARTER
4. SHALIAH BENTON
5. ANDREW KIM
6. KRISTIAN BERHOST
7. NAHEEM GRANT

Side note, these users have no tasks assigned to them

1. CHANDRA TYLER
2. MICHAEL THERIOT
3. NAHEEM GRANT

puts users.select { |user| Task.open.where(assigned_to: user).blank? }.map(&:full_name)

[araposo-tistatech]

Hi @hschallhorn Lazette Clanton is all set to have the feature enabled on Tuesday. Let's work together to schedule a time to turn it on.

[araposo-tistatech]

Hi @hschallhorn Lazette is out of the office today, but Ronald is available, let's setup the testing for him instead.

[hschallhorn]

@araposo-tistatech department-of-veterans-affairs/appeals-deployment#2747 Will enable this for Ronald as soon as it merges. Do we have a time he'd like this turned on?

[araposo-tistatech]

@hschallhorn 1pm please.

[hschallhorn]

@araposo-tistatech department-of-veterans-affairs/appeals-deployment#2762 Will enable this for Lazette when we hear back on when she'd like to try it out!

[hschallhorn]

Thinking out loud.
Attorneys and judges cannot use pagination because legacy tasks are not persisted in our database (which means they cannot be sorted and filtered at the database level on the back end). HOWEVER, legacy tasks will never appear in a users on hold or completed tab as we can only pull legacy appeals from vacols that are currently assigned to the user. Could we implement pagination for those two tabs only?

[hschallhorn]

Looking for attorneys to test!

attorney_ids = AttorneyTask.pluck(:assigned_to_id).uniq
attorneys = User.where(id: attorney_ids)
assigned_to_counts = Task.incomplete_or_recently_completed.where(assigned_to: attorneys).group(:assigned_to_id).order(:count).count
assigned_by_counts = ColocatedTask.open.where(assigned_by: attorneys).group(:assigned_by_id).order(:count).count
total_counts = assigned_to_counts.merge(assigned_by_counts) { |id, assigned_to_count, assigned_by_count| assigned_to_count + assigned_by_count }.sort_by { |id, count| count }.to_h

pp total_counts.last(10).map { |id, count| ["#{User.find(id).css_id}: #{User.find(id).full_name}", count] }.to_h
{"VACOCHOC: CYNTHIA CHO"=>53,
 "VACOTEMPLB: BLAKE TEMPLE"=>54,
 "VACOMORRAS: SHABNAM MORRAD"=>58,
 "VACOSCHICS: SUZANNE SCHICK"=>59,
 "VACOZHENGA3: ANDREW ZHENG"=>62,
 "BVARWATKINS: ROBERT WATKINS"=>65,
 "VACOALHINM: MOHAMMAD ALHINNAWI"=>66,
 "VACOTROTTR: RENEE TROTTER"=>71,
 "BVASBOEHM: SHAUNA WATKINS"=>80,
 "VACOKOMINB: BENTON KOMINS"=>102}

[araposo-tistatech]

@hschallhorn below are the attorneys we are approved to test with and schedule for enabling the feature:
Robert Watkins - 6/03 at 10 am EST
Blake Temple - 6/04 at 10 am EST
Benton Komins - 6/04 at 10 am EST

[hschallhorn]

We'll probably need at least until next week before this work for attorneys in done. Still working on priority stuff for this sprint before I can get back to this ticket!

[hschallhorn]

@araposo-tistatech Updated list if we want to pull the heaviest users now that we're ready to test!

attorney_ids = AttorneyTask.pluck(:assigned_to_id).uniq
attorneys = User.where(id: attorney_ids)
assigned_to_counts = Task.on_hold.or(Task.recently_completed).where(assigned_to: attorneys).group(:assigned_to_id).order(:count).count
assigned_by_counts = ColocatedTask.open.where(assigned_by: attorneys).group(:assigned_by_id).order(:count).count
total_counts = assigned_to_counts.merge(assigned_by_counts) { |id, assigned_to_count, assigned_by_count| assigned_to_count + assigned_by_count }.sort_by { |id, count| count }.to_h

pp total_counts.map { |id, count| ["#{User.find(id).css_id}: #{User.find(id).full_name}", count] }.last(10).to_h
{"BVASSHOREMAN: SCOTT SHOREMAN"=>45,
 "VACOSMITHA2: AARON SMITH"=>47,
 "BVATDOUGLAS: THOMAS DOUGLAS"=>48,
 "VACOKETTLR: RICHARD KETTLER"=>48,
 "VACOZHENGA3: ANDREW ZHENG"=>49,
 "BVAMTHOMAS: MEGAN THOMAS"=>54,
 "BVASBOEHM: SHAUNA WATKINS"=>60,
 "VACOSCHICS: SUZANNE SCHICK"=>64,
 "VACOKOMINB: BENTON KOMINS"=>80,
 "BVABBARON: BRITTANY BARON"=>99}

[araposo-tistatech]

Please enable this feature for tomorrow at 10am for the following users:

Andrew Zheng - VACOZHENGA3
Shabnam Morrad - VACOMORRAS
Blake Temple - VACOTEMPLB
Benton Komins - VACOKOMINB
Robert Watkins - BVARWATKINS

[araposo-tistatech]

Initial pilot feedback:
Robert Watkins - "I don't really see the difference in my account. Generally it is well organized and easy to navigate."
I inquired further with this user to see if he can see a difference in load times and he stated he does not. "It seems about the same to me."

[araposo-tistatech]

@hschallhorn (pinging you since I know you are keeping a close watch on this one), Andrew Zheng has provided great feedback, but has also flagged a bug that I am unsure existed before enabling this. He stated that on the "On Hold" tab the issues column is displaying all cases have only one issue in them even though he knows there are more issues in the cases. So far he is the only user reporting this.

[hschallhorn]

attorneys = User.where(css_id: ["VACOZHENGA3", "VACOMORRAS", "VACOTEMPLB", "VACOKOMINB", "BVARWATKINS"])
assigned_to_counts = Task.on_hold.or(Task.recently_completed).where(assigned_to: attorneys).group(:assigned_to_id).order(:count).count
assigned_by_counts = ColocatedTask.open.where(assigned_by: attorneys).group(:assigned_by_id).order(:count).count
total_counts = assigned_to_counts.merge(assigned_by_counts) { |id, assigned_to_count, assigned_by_count| assigned_to_count + assigned_by_count }.sort_by { |id, count| count }.to_h
total_counts.map { |judge_id, count| [User.find(judge_id).css_id, count] }.to_h
=> {"VACOTEMPLB"=>29, "BVARWATKINS"=>41, "VACOMORRAS"=>42, "VACOZHENGA3"=>51, "VACOKOMINB"=>78}

Most interested in Benton's feedback

[hschallhorn]

user = User.find_by(css_id: "VACOZHENGA3")
queue_config = QueueConfig.new(assignee: user).to_hash
tasks = queue_config[:tabs][1][:tasks]
tasks.map { |task| task[:attributes][:issue_count] }.uniq
=> [1]

Looking into this now!

looks like this is happening for all legacy cases, reproduced in dev as well

task_ids = tasks.select {|task| task[:attributes][:docket_name] == "legacy" }.map{ |task| task[:id] }
appeals = Task.where(id: task_ids).map(&:appeal)
appeals.map { |appeal| appeal.issues.count }
=> [2, 3, 4, 3, 1, 3, 1, 6, 2, 2, 2, 2, 1, 5]
tasks.select {|task| task[:attributes][:docket_name] == "legacy" }.map { |task| task[:attributes][:issue_count] }
=> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
CachedAppeal.where(appeal_type: LegacyAppeal.name, appeal_id: appeals.pluck(:id)).pluck(:issue_count)
=> [2, 1, 3, 3, 5, 1, 4, 3, 2, 2, 2, 6, 2, 1]

How are we determining issue count in queue config? https://github.com/department-of-veterans-affairs/caseflow/blob/aea0b24b8e570af5b5548a1374d82cc2ca64f7ff/app/models/serializers/work_queue/task_column_serializer.rb#L73-L79

appeals.map { |appeal| appeal.number_of_issues }
=> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
appeals.map { |appeal| appeal.case_record.case_issues.length }
=> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

How were we determining issue counts for legacy appeals? https://github.com/department-of-veterans-affairs/caseflow/blob/ba8b8ad060d7d37c1c38a7df1eefd0fc41c411b6/app/models/serializers/work_queue/legacy_task_serializer.rb#L83-L85

appeals.map { |appeal| appeal.undecided_issues.count }
=> [2, 3, 4, 2, 1, 3, 1, 6, 2, 2, 2, 2, 1, 2]

[araposo-tistatech]

@hschallhorn more feedback provided:

B. Temple:

1. The loading time navigating to the queue was fine. I did not notice any time lag compared to normal loading.
2. The sorting and filtering of the on-hold cases worked without any problem.

Any further feedback we need here? It seems as though there is not much impact for attorneys.

[hschallhorn]

No news is good enough news for me! happy to release whenever!

[araposo-tistatech]

@hschallhorn confirmed with the Board we are all set to release for tomorrow.

[hschallhorn]

Released weeks ago, no reported issues thus far