Amplify-js: Storage - more than 1000 keys

[marksteele]

Is your feature request related to a problem? Please describe.
There are many use cases which involve listing out a more than 1000 keys, however the storage api provides no means of doing so (nor any work-arounds that I could find).

Describe the solution you'd like
A mechanism to list out keys supporting paging/offsets/prefixes and delimiters.

Describe alternatives you've considered
Would also be possible to do with the raw S3 client if that was accessible from the Storage api.

Additional context

[harrysolovay]

Hi @marksteele –– I'd recommend creating a custom storage adapter for IndexedDB.

Storage adapters abide by the following interface.

interface Storage {
  // set item with the key
  static setItem(key: string, value: string): string;
  // get item with the key
  static getItem(key: string): string;
  // remove item with the key
  static removeItem(key: string): void;
  // clear out the storage
  static clear(): void;
  // If the storage operations are async(i.e AsyncStorage)
  // Then you need to sync those items into the memory in this method
  static sync(): Promise<void>;
}

And can be provided via a storage field in your Amplify config.

In auth, for instance:

import {Auth} from "@aws-amplify/auth";
import Config from "./aws-exports";
import {MyStorageAdapter} from "./wherever";

Auth.configure({
  ...config,
  storage: new MyStorageAdapter,
});

[marksteele]

Why would I want to use indexdb? The reason I'm using the Storage API is to leverage S3.

I have users who want to store thousands of files in S3. I'd like to be able to leverage Amplify (makes handling the authentication/token handling bits so simple, it's great!). I understand the reasons for dumbing down the API for some use cases, but to cripple it to the point where it removes some pretty basic functionality seems like a very odd design choice (nobody thought: "what if a user has more than 1000 files?).

Why can't additional functionality be added to the list function to expose more of the underlying S3 API? Is there some fundamental reason you can just add all the options from ListObjectsV2 to the Storage.list calls?

ex:

Storage.list('photos/', { level: 'private', ... other options .., listv2: {
ContinuationToken: 'asdf',
  Delimiter: 'asdf/',
  MaxKeys: 1000,
  StartAfter: 'STRING_VALUE'
}
}})
// Prefix would be auto-derived from access level and prefix passed in
/* returns something like
[{
key: 'photos/foo',
<other current keys>
Istruncated: true,
NextContinuationToken: "1w41l63U0xa8q7smH50vCxyTQqdxo69O3EmK28Bi5PcROI4wI/EyIJg=="
}]
*/

You could add some more properties to each returned item (to pass back the truncated flag and continuation token) which wouldn't be a backward-breaking change (not elegant, but that's the trade-off of possibly over-simplifying it).

Alternatively, when listv2 is passed in, return the raw S3 response back out and avoid the ugliness above.

[harrysolovay]

@marksteele apologies––I must have misread your initial comment. I thought you were referring––not to the storage category––but to underlaying storage mechanisms for things such as authentication state. I'll communicate about this with a team member who is more well-versed.

[kmbro]

I am looking for this feature as well. once you hit 1000 files or whatever your max page size is, you should get a key back. It is pretty basic pagination 👍

[boston99green]

Following this thread. I'm trying to retrieve all files in S3 bucket using React Amplify, It just list 1000 objects , but I do have more than 1000 objects in it. How can I retrieve it, Looking for a solution..?