Wp-calypso: (89P) Media Library: Account for legacy plans in max usage

[obenland]

As mentioned by @davemart-in in https://github.com/Automattic/wp-calypso/issues/43783#issuecomment-658740988 & https://github.com/Automattic/wp-calypso/issues/43785#issuecomment-658741018, for Business and eCommerce customers who signed up prior to August 2019 we need to make sure that the 200GB upload quota does not apply and they do not see any upload error messages.

There is a filter in get_space_allowed(), I wonder if that could be used to account for that special case.

[cpapazoglou]

👋 @davemart-in can we just clarify the following?

for Business and eCommerce customers who signed up prior to August 2019

I believe we should take into account for blogs created (or upgraded) on Business or eCommerce plans prior to August 2019 AND are still on Business or eCommerce plan.

[davemart-in]

@jmdodd would you be able to help me clarify this for Harris?

[davemart-in]

I spoke with Jennifer. Here are the notes from our convo:

• No matter what we can't have sites on Atomic infrastructure with > 200 GB. There are no exceptions.
• The over 200GB upload quota exception applies only to sites on our Simple architecture.
• ~This August 2019 clause should apply to any account that signed up prior to August 2019 AND happens to currently be on a Business or eCommerce plan. So, if they signed up prior to August 2019 and upgrade to the Business or eCommerce plan tomorrow, they should still see this exception.~ See @jennyzhu's reply.

And let me just double check with @jennyzhu that this is correct.

[jennyzhu]

@davemart-in @cpapazoglou Great question! It'd be what Harris said: blogs created (or upgraded) on Business or eCommerce plans prior to August 2019 AND are still on Business or eCommerce plan.

[davemart-in]

It'd be what Harris said: blogs created (or upgraded) on Business or eCommerce plans prior to August 2019 AND are still on Business or eCommerce plan.

I stand corrected!

[mreishus]

Note:

• The media storage endpoint for WPCom simple + atomic customers has its own layer of logic one layer on top of get_space_allowed(): It caps the max storage at 200GB and sets unlimited to always be false.
  Sounds like we should keep this for atomic, but either remove or modify it for simple business + ecommerce.

[mreishus]

From my initial research I see a few different paths we can take:

• (1) Filter get_space_allowed() to look for sites with the grandfathered plan and override the value returned.
• (2) Do a one-time update pass that finds the sites with the grandfathered plan and sets the option blog_upload_space to 1TB or larger.
• (3) Do a one-time update pass that finds the sites with the grandfathered plan and sets the option upload_space_check_disabled.

The one-time updates seem a bit cleaner, but I'm concerned about what would happen if one of these sites later moved to AT. I'm not certain if these options get reset on transfer, or how difficult it would be to change if they don't. Do you have any opinions about these different approaches, @jmdodd?

[mreishus]

WIP For Approach (1): D48741-code

[mreishus]

I've updated D48741-code with some important notes about the implementation. We want to align our approach with the previous changes that added the 200GB cap, to make sure that everything is happening on the same level.

[mreishus]

This has been modified to handle the unlimited cap and the 200GB cap in one plugin, while also simplifying the media storage endpoint a little.

[mreishus]

• D48741-code : Current code that works in real time with a 24 hour cache.
• D48938-code : First draft of a one-off script to add a sticker for blogs with legacy plans.

If we go with the sticker approach, then we either (1) Need a plan to remove the stickers when subscriptions lapse or are downgraded, (2) Need to change the D48741-code to check for current subscriptions in addition to sticker (not much code will be dropped) or (3) Give stickered blogs unlimited space for life, even if their subscriptions are canceled.

[jmdodd]

It would be helpful to know how many blogs qualify for this unlimited space (regardless of if they are Simple or Atomic at the moment), which should be a pretty simple query. The more complicated this becomes, the harder it will be to maintain.

[mreishus]

Overview

@jmdodd and I took some time to investigate; we've decided an approach different than my two patches above is more appropriate. The goal is to bring the data in the db as close to reality as possible while minimizing the number of hacks obscuring that data.

New eCommerce and Business plans have incorrect storage data in DB (2TB)

They're provisioned with blog_upload_space = 2TB, but they're only supposed to have 200GB available. The media storage endpoint returns 200GB for these sites, but they still have 2TB as the value in their database, which other parts of the system might read. Reporting could be affected as well.

_(Prior work: D40523-code)_

2TB is treated as 'unlimited' throughout the code

Now we have a problem. eCommerce and Business plans created before 2019-08-01 have blog_upload_space = 2TB, and they should be unlimited. Those created after also have blog_upload_space = 2TB, but they should be 200GB.

How do we know which 2TB values are really unlimited, and which are really 200GB?

• _Abandoned Approach 1_: Whenever looking for available space, do a costly lookup of previous subscriptions to see which time window they fall in, so we know how to treat the 200GB value. D48741-code
  
  
  
  • What used to be a simple DB lookup is now much more complex, and slow. Other parts of the system might not honor this calculation.
  
  
• _Abandoned Approach 2_: Do a one-time pass to add a legacy-plan sticker to eCommerce and Business plans created before 2019-08-01. When running into a 2TB value, check if the sticker exists. Depending on the sticker, we'll know if the 2TB really means "unlimited" or if it means "200GB". D48938-code
  
  
  
  • Workable, but it seems like a bad decision to have to meanings for the 2TB value that must be disambiguated elsewhere.
  
  

Plan going Forward

In the DB, 2TB blog_upload_space will always mean unlimited space instead of having two different meanings, and 200GB blog_upload_space will always mean 200GB.

Steps to Implement

1. Modify provisioning so new eCommerce and business plans are created with blog_upload_space = 200GB.

2. Do a one-time pass to update the blog_upload_space for eCommerce and business plans created after 2019-08-01 to = 200GB.

   2a. It's possible that some of these blogs have found a way to upload more than 200GB, and will be upset that they can no longer upload images after we adjust this down. We can check to see how many of these there are, if any, by doing a one-time iteration with _get_blog_disk_usage() with $rescan=false.

3. Simplify the media endpoint code. It shouldn't reference a 200GB cap anymore. (See: the version in D48741-code)

4. Add a simple filter to get_space_allowed which caps to 200GB for Atomic sites. (This is an atomic infrastructure decision, not plan related. Additionally, blog_upload_space doesn't do anything for Atomic sites).

This is fixing it "the right way" and bringing the database as close as possible to the reality of the storage allowed.

(cc: Product[ @davemart-in @amamujee ] Store[ @rralian ] Legal[ @jennyzhu ])

[mreishus]

Keeping track of new work being done in this note.

• Check to see how many post-legacy simple ecommerce+business blogs are currently using over 200GB.

  • Added a crawler to check here: D49237-code (Needs Review)

• When sites are transferred from Simple -> Atomic, their blog_upload_space should be modified upward to 2TB.

  • Possible code location: 2660a-pb
  • ( Jetpack sites get 2 TB for VideoPress. When a site is Atomic, the space used has pretty much zero bearing on anything except VideoPress uploads. )

• When sites are transferred from Atomic -> Simple, their blog_upload_space should be modified downward to 200GB.

  • Possible code location: 2660a-pb

• Modify provisioning so new eCommerce and business plans are created with blog_upload_space = 200GB.

  • Business: D49143-code (Needs Review)
  • ECommerce: D49149-code (Needs Review)

• Do a one-time pass to update the blog_upload_space for eCommerce and business plans created after 2019-08-01 to = 200GB.

  • TBD

  • It's possible that some of these blogs have found a way to upload more than 200GB, and will be upset that they can no longer upload images after we adjust this down. We can check to see how many of these there are, if any, by doing a one-time iteration with _get_blog_disk_usage() with $rescan=false.

  • Added a crawler to check here: D49237-code (Needs Review)

• Simplify the media endpoint code. It shouldn't reference a 200GB cap anymore. (See: the version in D48741-code)

  • TBD
• Add a simple filter to get_space_allowed which caps to 200GB for Atomic sites. (This is an atomic infrastructure decision, not plan related. Additionally, blog_upload_space doesn't do anything for Atomic sites).
  
  
  
  • TBD
  
  

2020-09-11 update: paYJgx-Zu-p2

[rralian]

In the DB, 2TB blog_upload_space will always mean unlimited space instead of having two different meanings

This is not a blocker, but that still means "2TB" means something other than 2TB.

My other thought is that a single product should not represent two different sets of features. The cat seems a little out of the bag at this point. But if we run into things like this in the future, we should really differentiate the product we are selling and keep things separate that way.

[mreishus]

Progress here has been tracked in: paYJgx-Zu-p2

We've changed new and business ecommerce plans to be provisioned with 200GB of space instead of unlimited. What remains is a one-time update of plans in the last ~14 months, and simplifying some of the wpcom endpoint code that's no longer needed. Those have been split into two issues you should see crosslinked above.