Vscode-extension-for-zowe: Support Large Datasets and PDS Members

[jalel01]

Objective: Test to ensure UX is the same for "large" files as for small / average files.

We have a few open questions with regard to support for large datasets in Zowe Explorer:

• Need to define what we mean by "large" in terms of size of a single dataset?

• Should we also take account of support for interacting with multiple datasets which, taken together, add up to "large"? 

• Need to determine what functions need to be supported? What actions specifically are we talking about in Zowe Explorer?

Once we understand the above, our objective will be to validate behaviour and log any issues we find.

Happy to hear from anyone with thoughts on the above open questions!

[venkatzhub]

I will take a stab at answering some of the questions:

1. Large datasets - in reality large number of datasets: This scenario is when a user types in a filter (e.g HLQ.* ) - and the expansion of that results in 20k - 40k datasets being displayed in the UI. The "large" in terms of size is hard to define - one approach can be to research the current behavior starting with 5k, 10k,15k,20k, upto 40k and see where things start to break or become 'slow' - i.e find the threshold first.
2. Related is large number of members in a single dataset - typically I have seen 40k-50k members in a single dataset. The scenario can be a simple one where the user searches for a dataset, and expand that dataset.
3. The functions supported should be the same as what are provided on standard datasets and members (allocate/create, copy, paste, rename, delete etc.)

[venkatzhub]

The Core problem statement is - how Zowe explorer manage large number of datasets or large number of members in a given dataset with reasonable performance and user experience?

[jellypuno]

Initial Research:

When trying to view large number of dataset (e.g. *.PUBLIC.*)

• Zowe Explorer has a timeout of 30 seconds. If the loading takes more than 30 sec, it will display an error

• Zowe Explorer is trying to issue HRECALL in every dataset that you are trying to access (This is because of the z/OSMF header). An error will display if you are using CA DISK.

• If you are successful in viewing the files, it will look like this

Issue #172 will allow you to filter the filename that you want to search if the list is too long.

Notes:

• Points 1 and 3 will also apply when viewing large amount of PDS members.
• If you have successfully viewed the files, you can use Zowe Explorer as normal (e.g Edit, Delete, Rename, Submit etc.)

[jellypuno]

I was able to get a PDS with 30k members. These are the results:

1. Searching for the dataset: 3 secs
2. Opening the PDS to see the 30k members: 5secs
3. Opening the member: 2 secs
4. Saving a modified version of the member: 8 secs
5. Downloading anew copy of the member: 1 sec

Viewing all the members is still a similar experience in point#3 in the previous comment. Issue #172 will really help in this situation.

[jellypuno]

Copy and Rename takes more than 5 minutes to re-load. You cannot do anything while it is loading. The only thing left to do is restart your VSC workspace.

[jelaplan]

@jalel01 referenced a pagination mechanism for handling large numbers of members. Has that been planned?

@jellypuno Is there some plan to handle 172 where the UI includes a way to filter within a dataset to narrow down the list of members?

[jellypuno]

@jelaplan For Issue #172 , I believe the approach is to filter

• the list of dataset names that you have if you searched using HLQ1.*
• the list of member names inside a dataset.

@katelynienaber Please correct me if I am wrong.

[jalel01]

Thank you @jellypuno for your presentation today of the research outcome. For reference it can be found here: https://ent.box.com/s/nxnde5isjgv2czdc9ch2z7f3zf81n41v

@jelaplan @RASakach next step is to refine and then plan suggested enhancements. Including the pagination mechanism.