Fec-cms: Break out issues that are left that are blocking the wagtail and Django upgrades

[PaulClark2]

THIS IS AN INFORMATIONAL ISSUE THAT CONSOLIDATES OUR KNOWN ISSUES

What we're after:

Initial testing of Wagtail 2.0 has revealed some issues that are dealbreakers that must be fixed:

• Mandatory sections in the resource page and custom page templates (for example, the asides, the citations block and the record article links) need to be optional. Not every page needs them, and we can’t make changes to existing pages that don’t use them if this isn’t fixed. (For example, the careers page and the pending rulemaking matters page.)
• Bulleted lists in collection page template don’t show the bullets except for the first one.
• Document lists don’t generate in the document feed template and same with the document page.
• We really do need the html editor. We use it to move things from feature into production. It also can serve as a workaround for some of the rich text editor bugs. Plus, it’s just an expected thing that something will turn up that we need a html editor to fix.
• The glossary button in the rich text editor is problematic. Glossary button only works if the term is exactly like it is in the glossary. (That is not usually the case; for example, a word like contribution might be pluralized, or a phrase like “coordinated communication” might be put as “communication that is coordinated.”) While editing can serve as a work around, we would need html editor ability to code glossary to keep it from looking funny.

Issues relating to [something about Python] and things that need fixing in the rich text editor (but that aren't dealbreakers) will be in separate tickets. The RTE editor issue is #2024 .

Background:

Initial testing of Wagtail 2.0 page templates and notes: #1973
Write-up of dealbreakers and issues in Rich Text Editor found during initial testing of all templates: https://docs.google.com/document/d/1fgxx9oTEXunm7scqyqJRDF5bG-z45eXO1VkOLwSa8Zw/edit?usp=sharing
Related Django and Wagtail updating issues: #1538 and #1968 and #2011
Wagtail 2.0 and hard coded pages #1994
HTML editor issue #1971 and #2024
Python tests failing #1972
The content team's Wagtail wish list: https://docs.google.com/document/d/1HM1-SaxAL-pVhzwQY8ZP-1uA8bVlbAxwFhqW4f-3lAo/edit

Completion criteria:

• [ ]

Annalee has capacity to work on Django issues, tagging @annalee
Difficulty - 10 points

[llienfec]

We need to divide up this ticket into several. @pkfec is willing to help!

[dorothyyeager]

I've updated the start of it with the background info I had and references to the other background tickets.

[dorothyyeager]

New ticket written up for the RTE issue and I've edited this issue with the dealbreakers. Now we just need a ticket written up regarding the python issue that affects this.

[llienfec]

One thing that @dorothyyeager and I really need is ability to use anchor links - whether in the editor OR in HTML. It looks like Wagtail is currently enabling this functionality and the work is tracked here:
https://github.com/wagtail/wagtail/issues/1049

Not sure if something in their discussion is helpful or if you need us to reach out to their team about a time frame. Happy to do it.

[AmyKort]

Let's plan to get everyone together during this sprint to figure out what Wagtail issued get tackled when and by whom.

[dorothyyeager]

Adding one more item to the mix. Not a dealbreaker, but something @johnnyporkchops suggested I test - scheduled publishing of a page - failed. The page did not go live when I had set it to go live, and I had to manually publish it. Am going to test expiry of the same page tonight (to see if it comes down when I scheduled it to come down).

[johnnyporkchops]

@dorothyyeager , this feature works (at least for dates but not times) on the current version of wagtail we are using. Flagging that we might want to test this again once when we get further along with 2.0 as it might just be a settings issue.

[dorothyyeager]

And just to note that the page did not expire at the time it was set to expire. Will test again as suggested above when we are further along.

[fec-jli]

for Python test, IMO:
1)Can we make sure Django.test TestCase(built-in Django) work first after upgrade Django to 1.11.13?
2) make pytest work (use pytest-django package ??)
3) think about upgrade Django to 2.0.1
4) let Django.test and pytest both work in fec-cms.
any suggestion?@annalee Thanks

[annalee]

@fec-jli As I understand our priorities at the moment (correct me if I'm wrong!), we want to:

• Upgrade Wagtail and Django, which requires:
• Getting the tests working, which requires:
• Dealing with the issue with our database migrations.

We're trying to tackle the underlying migrations issue head-on rather than move to PyTest to work around it, because:

• The underlying issue is a source of technical debt. Working around it is time-consuming, and the longer we work around it, the harder it will be to fix.
• The pairing we did this week revealed that changing the test execution order doesn't actually fix the problem, so at this point we're not sure that moving to PyTest will actually fix it.
• Fixing the migrations de-couples move to pytest from upgrade wagtail, making them to separate issues that don't have to be addressed together.

So our current plan of action is:

1. Fix the migrations issue. This will fix Django's test framework.
2. Write a few catch-all tests in Django's test framework that will tell us if we manage to massively break things.
3. Upgrade Wagtail and Django, using our new tests to make sure we haven't massively broken anything.
4. Move to PyTest (starting with moving our existing tests into pytest).

As far as:

4) let Django.test and pytest both work in fec-cms.

@fec-jli can you talk a little more about why we want them both running for fec-cms? The two frameworks can exist side-by-side without causing any problems, but it will be easier to maintain and run our tests if we keep all our tests in one framework or the other.

I personally agree with @patphongs that it makes sense to move to PyTest, since the API is already using it, and because it has more robust features. However, we have a great opportunity here to set up our testing processes from the ground up, so if there are things you prefer about Django's built-in framework, we should talk about it and make sure we're setting up a test process that incorporates everyone's feedback.

[patphongs]

@annalee @fec-jli @rjayasekera @ccostino Thank you for working on this issue. There are 2 separate issues right now in 6.1. I am not sure if it all can be accomplished, please let us know if we should revisit this.

1. #2011 Fix Migrations and Switch to PyTest from Django
2. #2006 This current issue to fix dealbreaker wagtail 2.0 issues.

I have made these two issues dependencies before we can merge the upgraded wagtail and django PR https://github.com/fecgov/fec-cms/pull/1968

[annalee]

@patphongs What do you think about splitting #2011 into two issues, since switching to PyTest is not a blocker on upgrading Wagtail, if we're fixing migrations separately?

[fec-jli]

@annalee Thank You for your explanation, I just join this ticket recently. may not know more detail for the progress.
I agree with your plan:
1)Fix the migrations issue. This will fix Django's test framework. 2)Write a few catch-all tests in Django's test framework that will tell us if we manage to massively break things. 3)Upgrade Wagtail and Django, using our new tests to make sure we haven't massively broken anything. 4)Move to PyTest (starting with moving our existing tests into pytest).
question for 1), after fix migrations issue, will fix Django's test, that is just what I want:)
question for 3), which version will we upgrade finally? Django 2.0? Wagtail 2.0.1 ? right?

after we done with 4), both Django's test feature and py.test can be used in fec-cms, right? I am not mean we will use both. sorry about confusion.
Thank you so much.

[annalee]

Ah, yes, @fec-jli--sorry for the confusion. Both frameworks will be able to run against the tests when we're done.

[patphongs]

@annalee I agree! Let's split out switching to PyTest and fixing the migrations.

[annalee]

Those are now split into #2040 and #2041, with #2040 in 6.1 and #2041 sitting in "new issues" to be groomed into a later sprint.

[dorothyyeager]

Dealbreakers/Must fix before initial use:

• Make mandatory sections in the resource page and custom page templates optional:

  • [x] the asides
  • [x] the citations block
  • [x] the record article links

• Collection page template -- Make all bullets in bulleted lists show. Currently only first bullet shows.

• Document lists don’t generate in the document feed template and same with the document page. John is working on this in #1994 .

• Add html editor.
• Fix glossary button - see #2024

Details are in the content team google doc at https://docs.google.com/document/d/1fgxx9oTEXunm7scqyqJRDF5bG-z45eXO1VkOLwSa8Zw/edit?usp=sharing).

[apburnes]

Reviewed latest editor updates and deal breakers for wagtail upgrade with content team.

__Final ToDo__

• [x] Add terminology select to Glossary button

[apburnes]

Closing because remaining deal breakers are addressed in #1968