Aws-sdk-js: split aws-sdk-js into API-specific modules

[andyburke]

Hi,

I'm trying to build some lightweight docker images and if aws-sdk is ever included it adds almost 30MB of code into node_modules. (Related: #1748)

It seems likely that many consumers of this module only use a handful of AWS APIs, especially with the rise of focused microservices and serverless approaches.

I usually prefer to open PRs addressing an issue rather than an issue itself. However, this kind of change would need a lot of discussion, and I certainly wouldn't be in a good position to implement such a change, having no experience with the internals of the library.

At a high level, I would imagine something like a series of modules:

aws-sdk-core (shared core libraries, hopefully relatively small)
aws-sdk-s3
aws-sdk-ec2
...

aws-sdk-js could have dependencies on all of them, maintaining backwards compatibility for those who are not sensitive to the install size and just want to npm install aws-sdk.

For the space-conscious, we could install only the necessary APIs, eg:

npm install aws-sdk-s3

Since the clients are already separate, maybe this would not be as large of a change as I imagine it might be? Unfortunately, I'm not at all familiar with the internals of this library, so I have no real understanding of how feasible a change like this would be, but it would definitely be helpful for me if I could get slimmer, API-specific installs in my images.

Thanks!

[jeskew]

Hi @andyburke,

I hear and share your concern, but I'm not sure this could be done in a backwards compatible way. We encourage users to import submodules of the SDK in our official documentation, so the SDK's current disposition on disk is effectively a public API that we can't change without a major version bump.

After running npm pack on the SDK to see the source of our package size, it looks like the primary source of bloat on disk is the dist folder (13.7 MB on disk, tracked in #1748) and the clients folder, (12.2 MB on disk, 11.6 MB of which is TypeScript definition files).

[thieman]

In addition to size on disk, this issue also contributes significantly to TypeScript compile times, as you're forced to load type definitions for the entire project (in the neighborhood of 200K lines!) on every compile even if you are only using a specific module.

[tristanhoy]

I faced exactly the same issue deploying to Lambda, and managed to trim [email protected] down to 6.44 MB with the following script.

const fs = require('fs-extra')
const glob = require('glob')
const path = require('path')

function remove (pat) {
  glob(path.join(__dirname, pat), (err, files) => {
    if (err) console.error(err)
    else files.map(file => fs.removeSync(file))
  })
}

remove('/**/aws-sdk/dist')
remove('/**/aws-sdk/dist-tools')
remove('/**/*.d.ts')

Because this script makes assumptions about the internal structure of the module, you'll need to test/adjust it whenever you upgrade aws-sdk (and make sure you use pinned versions)

[srchase]

@andyburke

I just wanted to chime in that v3 of the SDK is available now as a preview.

It has a modular design and achieves what you were requesting in this issue.

I'm going to close out this issue as v3, though not production ready just yet, is where we'll be focusing our work to meet this need for the SDK.

[springmeyer]

For those that are curious, it looks like v3 development is at https://github.com/aws/aws-sdk-js-v3

[srchase]

@springmeyer

Thanks! I should've included the link to that repository.

There's a blog post with additional details here: New AWS SDK for JavaScript – Developer Preview.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.