Zulip: export: Ensure zulip->zulip export covers all tables that should be exported

[timabbott]

I just completed the TODO project we'd for a long time had at the top of ALL_ZERVER_TABLES (well, now it's ALL_ZULIP_TABLES) to validate the list of tables that we export against the list of tables that are actually present; see 42aea68df3cc9e58c8e8ac857aab5bc36022129a. I suppose this probably increases the priority of making sure we call sanity_check_output in CI (or at least refactor so the first piece is run there).

Aside from that bit of tooling work, the TODO list of things to fix is this:

    # The fact that these are not exported is a bug
    'zerver_botstoragedata',                                                                                                             
    'zerver_botconfigdata',                                                                                                              
    'zerver_mutedtopic',                                                                                                                 
    'zerver_realmauditlog',                                                                                                              
    'zerver_pushdevicetoken',                                                                                                            
    'zerver_service',                                                                                                                    
    'zerver_usergroup',                                                                                                                  
    'zerver_usergroupmembership',                                                                                                        
    'zerver_userhotspot',                                                                                                                

A few of these are larger projects (like botstoragedata / botconfigdata), just because we may need to add some to populate_db to make testing it convenient. But I think most of these are pretty straightforward and we should just aim to fix quickly. Some notes:

• user groups are present in populate_db
• hotpots are present in populate_db
• muted topics are easy to make in the UI
• realmauditlog entries are easy to make, but this one make require a bit of work just because of its foreign key pieces.
• push device tokens might need to be made manually (see test_push_notifications.py).
• service comes from making an outgoing webhook or embedded bot

And then medium priority is to also take along the analytics tables:

    'analytics_fillstate',                                                                                                               
    'analytics_realmcount',                                                                                                              
    'analytics_streamcount',                                                                                                             
    'analytics_usercount',                                                                                                               

(Check out the docs at http://zulip.readthedocs.io/en/latest/subsystems/analytics.html for how to get a analytics.ds realm with good test fixture data for this)

[timabbott]

@rheaparekh FYI, this is the follow-up I'd mentioned :)

[rheaparekh]

@zulipbot claim

[zulipbot]

Hello @rheaparekh, you claimed this issue to work on it, but this issue and any referenced pull requests haven't been updated for 10 days. Are you still working on this issue?

If so, please update this issue by leaving a comment on this issue to let me know that you're still working on it. Otherwise, I'll automatically remove you from this issue in 4 days.

If you've decided to work on something else, simply comment @zulipbot abandon so that someone else can claim it and continue from where you left off.

Thank you for your valuable contributions to Zulip!

[rheaparekh]

@zulipbot I am still working on this.

[timabbott]

This is fixed via db1260fb939feaa01da4570d5bc0dc3f567b6e4b, though we haven't done the analytics tables (which I'm not sure are really important to do; worth doing if they're easy, I guess).