Middy: Defining a Roadmap for 1.0.0

[lmammino]

Hello all,
I think this project is starting to mature and slowly get a bit of traction.
I believe it's important that at this point we set a vision of what we want to achieve in a "first" stable version so that we can go out of the experimental phase and have a clear direction of what do we want to achieve with this project.

I would suggest starting by listing the pain points that currently exists and then missing features that would be needed in order to consider this project "production ready" (or a high-quality open source project if you prefer).

I am going to get the ball rolling by listing my pain points and missing features after this post.

I would kindly ask all the maintainers (and some of the users I know) to do the same exercise so that together we can build a shared vision and take into accounts different needs and use cases.

Please repeat points I already mentioned if they are important for you too.

CC: @dkatavic @vladgolubev @joseSantacruz @kevinrambaud @nnyegaard @KnupMan @peterjcaulfield @garyhollandxyz @roxeteer @miki79 @Juanelorganelo @rafaelcorreiapoli @i-am-kenny @NPCRUS @Jolg42 @DavidWells @acambas @theburningmonk @techfort

[lmammino]

Pain points (in order of importance)

• (httpHeaderNormalizer) Normalized headers are confusing. Most of the other JS frameworks just lowercase the headers, so it's a common expectations for users that the same happens in middy. We should probably reconsider this approach.
• Bundling all the middlewares with the core makes the project bulky, potentially slower and hard to maintain. We should probably re-consider the previously mentioned idea to keep the core in a dedicated package and every middleware in it's own package
• Typings and middlewares documentation are hard to maintain up to date

Missing features (in order of importance)

• ResponseSerializer (based on content negotiation) middleware
• Generic Error Logger middleware
• List good 3rd party middlewares as part of the documentation
• No "structured" changelog is maintained when every new version is published
• Missing guidelines and templates for opening issues and PRs
• Create a better website that makes clear what problem middy is trying to solve, what are the main features and a bunch of tutorials on getting started with the most common use cases (writing an API, a cron job or an S3 file processing lambda).

[roxeteer]

Before digging any further to the issues, I'd like to propose that in the future you wouldn't do as many releases as before. Each changeset shouldn't trigger a version update.

[KnupMan]

@roxeteer so maybe it's better to define the goals for versions or something like milestones?

[hollg]

I'd echo pretty much all of @lmammino's list with a particular focus on guidelines for contributing to the project and splitting middlewares out into their own packages.

[kevinrambaud]

Great news.

I agree with all the points that have been mentioned above especially :

• _Project structure_
  I followed the issue #78 related to project structure and I'm not we stated on a project structure. We know that keeping the middlewares into the core is not the way to do but in this case, do we go the monorepo way ? single repo by middleware ?
  I don't have enough feedback with monorepo to say this is the way to go, but in term of maintenance it looks really great.

• _Typing and middlewares documentation_
  Regarding typing, unfortunately, from what I know, except if we use completely typescript, I don't know a way to generate typing automatically based on pure javascript, and even if we were using the jsdoc it, I would not be confident of the typing generated as it would rely of the jsdoc we would write.
  Regarding middlewares documentation, a first approach could be to create a dedicated README per middleware if we go the monorepo way and explore possibilities to use these readme to generate docs on the website

• _httpErrorHandler middleware_
  From what I see of the current httpErrorHandler middleware, it is not yet possible to define the format of an error response. I think this is a required feature for the 1.0.0 as many companies rely on custom Error object that contains additional data or different format. We could keep the current implementation minimal with statusCode and body keys but offer the flexibility to define a custom format.

• _No "structured" changelog_
  It can handled by different ways, either manually (but who would like to do that !) or by using a tool like semantic-release, I recently integrated it on some projects and it's really amazing, changelog generation, commit message standardized, commit lint, version number management based on commit message, it's really flexible

• _Missing guidelines and templates for opening issues and PRs_
  +1, I would also add some example of use cases to create different kind of middleware to manage before, after and onError like basic implementation, it can sound trivial, but it can help to onboard new people especially newcomers to lambda

• _Create a new website_
  Fantastic, probably the most important part to succeed, what makes great library after its features is its website with documentation

[lmammino]

@roxeteer thanks for your feedback. I would like you to argument more your feedback though... What is that you don't like about having many releases?

Personally, I am a strong proponent of having a release per every change, so that every version is immutable and you can install the package everywhere/anytime with predictable results.
But maybe I am not seeing some important drawback with this approach.

Also thanks to @garyhollandxyz and @kevinrambaud for their feedback. Later on, I'll try to break down the results of this conversation into issues and create a milestone to track the progress.

[dkatavic]

Kudos to @lmammino for taking an initiative to create this thread.

I agree with all the points you made, especially with:

• Separating core and the middlewares
• List of 3rd part middlewares
• Structured Changelog
• Guidelines for PRs

Regarding releases/versioning, I personally like semVer approach and release regularly. As we are going to have a separate core and the middlewares, we should see fewer releases. I am open to any other suggestions regarding this

Regarding having a monorepo for all middlewares or repo per middleware, I like the Unix philosophy of minimalistic modular software development, so I am for creating repo per middleware. I believe that most of the NodeJS frameworks use the same approach (as express )

Do we want to use some tool to keep track of the work needed for V1 (Like GitHub projects)? One of the actions I would like to make is to create 1 repo with a simple middleware where we can agree how to structure middleware repo

[kevinrambaud]

@dkatavic regarding repositories structure (mono vs multi), I also really love the minimalistic module philosophy, on the other hand, it can be a pain in the ass to maintain repositories especially when we can end up having ton of repositories (having to subscribe to all repositories, manage issues, pull requests separately).

Some orgs like express and koa end up with ton of micro repo that are just not maintained or people don't see them because there is dozen of repos.

From what I see, monorepo such as Babel or React are doing well when managing repo, I see some benefits with their packages structure. Have you considered this possibility ?

[nnyegaard]

Pain points

• _Typing is not always update to date_
  I think the best way to keep the typing updated is to convert this project into Typescript. That being said I don't know how many people are using Typescript with middy, so that may not be the bedst for the project. I would not mind taking on the task of keeping the typing updated, since i'm using middy with typescript.

• _Bundling all the middlewares with the core makes the project bulky, potentially slower and hard to maintain_
  I like the way React/Bable/Others does it with a mono repo where they use Lena to help them. Since we have a core, but we also have other packages that are related having it all in one repo would help us maintain in but reduce the size of one package.

• _Examples_
  Having a folder with examples and best practices would be awesome, and help a lot of new people, I think!

Missing features

• Generic Error Logger middleware
• Generic logging for input and output before it hits and leaves handlers
  This could be part of a enable debug mode or something like that to get an understanding of what happens

[dkatavic]

@kevinrambaud I have never used lerna, I will definitely take a look. It looks like an interesting solution

[lmammino]

@nnyegaard I really love the idea of a debugMiddleware, it should also be quite easy to implement.

Regarding Lerna, I think we had a conversation already and it definitely looks like a great option. Still I haven't used it before, so I would have to experiment a bit with it to understand how it works.

Regarding typescript, thanks for volunteering, but I would try to avoid having a gatekeeper for this kind of things. Maybe we should enforce, as part of the PR process, people to make sure they update the typings (as long as the documentation). PR templates might help with that, as we can add a checklist like the following:

PR checklist

• [ ] I added tests for my changes
• [ ] I updated the documentation to reflect my changes
• [ ] I updated the Typescript typings to reflect my changes

[kevinrambaud]

I also think that the typescript typing should be handled manually. As you mentioned, by adding it as a requirement in the PR template, it should do the job.

Regarding debugMiddleware do you already have more details about what you're thinking about ? Is it something similar to debug package ?

[lmammino]

I am starting to open issues under the newly create milestone v1.0.0

[nnyegaard]

@lmammino @kevinrambaud I totally agree, that having a gate keeper/ bottleneck is not something we want, so if it's possible to get Typescript as part of the PR process that would be awesome!

[lmammino]

Just to make you aware, I finally managed to bootstrap this work using Lerna and we have now a 1.0.0 work in progress in the branch 1.0.0-alpha.

You can see packages already published in the middy org on npm.

I suggest we work on the outstanding issues for releasing 1.0.0 as PR to this new branch and only when all the work is done we can integrate with master and deprecate the old middy package.

I look forward to receiving your feedback :)