Polis: Conversation data export not working via UI

[urakagi]

Hi, I give up to make polisReport working and want to export data from my local polis instance.
But it's a pity that the export function is not working either. It shows no any error but I just don't have the mail.
Is there any hint or SMTP setup I need to do? Thanks.

To Do

^{todos added by maintainers}

• [x] temporarily remove data export from client-admin UI
• [ ] add mention to export page of knowledgebase
• [ ] document manual data export via command line (server administrator)
• [ ] fix data export UI

[urakagi]

I'm now generating csv file from database directly. Can I please have a hint that how I get group-id for participants-votes.csv? Thank you very much.

[metasoarous]

Hi @urakagi. We're aware that there's a issue with the export functionality right now, and I'm working on that. Sorry for the hassle.

I just pushed some changes to master that make it possible to export from the command line like so lein run export -f thefilename.zip -Z 2convurlid. You'll have to have all the same environment variables; Also note that this only exports from the prod math results buckets, so make sure you've been running the math worker and server with MATH_ENV=prod. If you need to export from the preprod or dev (or whatever other math env buckets), let me know and I'll add a flag for this.

[urakagi]

Thank you @metasoarous, I tried to run this and get this error:

18-10-31 01:35:49 chttl-b6a0edd74707c886.local.com INFO [polismath.runner:158] - Runner main function executed
18-10-31 01:35:49 chttl-b6a0edd74707c886.local.com INFO [polismath.runner:174] - Running subcommand: export
Exception in thread "main" java.lang.NullPointerException, compiling:(/tmp/form-init7699330447764564917.clj:1:73)
        at clojure.lang.Compiler.load(Compiler.java:7526)
        at clojure.lang.Compiler.loadFile(Compiler.java:7452)
        at clojure.main$load_script.invokeStatic(main.clj:278)
        at clojure.main$init_opt.invokeStatic(main.clj:280)
        at clojure.main$init_opt.invoke(main.clj:280)
        at clojure.main$initialize.invokeStatic(main.clj:311)
        at clojure.main$null_opt.invokeStatic(main.clj:345)
        at clojure.main$null_opt.invoke(main.clj:342)
        at clojure.main$main.invokeStatic(main.clj:424)
        at clojure.main$main.doInvoke(main.clj:387)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
        at polismath.system$create_and_run_system_BANG_.invokeStatic(system.clj:63)
        at polismath.system$create_and_run_system_BANG_.invoke(system.clj:61)
        at polismath.runner$_main.invokeStatic(runner.clj:175)
        at polismath.runner$_main.doInvoke(runner.clj:156)
        at clojure.lang.RestFn.invoke(RestFn.java:482)
        at clojure.lang.Var.invoke(Var.java:403)
        at user$eval61568.invokeStatic(form-init7699330447764564917.clj:1)
        at user$eval61568.invoke(form-init7699330447764564917.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7062)
        at clojure.lang.Compiler.eval(Compiler.java:7052)
        at clojure.lang.Compiler.load(Compiler.java:7514)
        ... 12 more

Then lein stops. I copied run script with the enviroment variables. MATH_ENV=prod. The main polisMath instance is running. What can be wrong?

[urakagi]

Also, what is the repo you pushed? I don't see any new commit on polisServer or polisMath.
I have completed to rewrite an exporting function other than group_id, so I'll be happy to have an explanation about how to get it.

[metasoarous]

My bad, I pushed to the wrong repo earlier. It's up there now if you pull down master again.

[urakagi]

@metasoarous It worked, thank you very much.
Also I have sent a PR https://github.com/pol-is/polisMath/pull/3, to write down this command as script file and clearify what environment variables need to be set.

[metasoarous]

I'm going to reopen this issue, as the underlying issue with polisServer isn't resolved yet.

[patcon]

Is data export issue still unresolved @metasoarous?

[patcon]

Colin confirmed in https://github.com/pol-is/polis-issues/issues/137#issuecomment-624907223 that it's not working at the moment.

Any pointers on where the issue might be?

[patcon]

Came up in chat when a newcomer was confused by mention of export in knowledgebase.

Gitter chat:

@arjentum
Also, I'm not sure if I'm missing something really obvious, but how does one export the data from a Pol.is report? I think I've looked through all of the conversation admin pages already...

@patcon
No no you're not crazy! @colinmegill rewrote client-admin lately, and we removed the export interface until someone has time to fix a bug that was preventing users from exporting: https://github.com/pol-is/polis/issues/39#issuecomment-434492966

@patcon
Also, we should probably add a prominent mention of this issue to the knowledge base.

We should add a mention of this in knowledgebase export page

[patcon]

More gitter chat

we're happy to provide you with exports for this project till we get that system fixed up, which we're aiming to do within the next few months.

[patcon]

https://gitter.im/pol-is/polisDeployment?at=5f651c0ef969413294f45771

Yes; To confirm what @joshsmith2 said, it is possible for us to get xids out in the exports, it's just not allowed in the UI currently.

To be clear, we don't automatically include xid in the exports, since they can be potentially sensitive (emails, etc).

I think we'll probably at least have an option for doing this though, and using personal information like emails directly for xid is probably a bad pattern anyway when it can be avoided.

[colinmegill]

Maybe we should break the data export up. Only some of the files are too big to send over the wire. We could have the summary, and comments, be more visible / explorable in browser as raw data and copy-able as CSV. The raw matrix could be too, up to a certain size...

Bowling Green, 2000 people and 1000 comments, is 2 mb

[colinmegill]

raw.pol.is/convoID_votes.csv?

[colinmegill]

Or pol.is/raw/convoID/votes.csv?

[patcon]

+1 from @hamnox in Gitter chat :tada: thx!

[patcon]

For future travellers looking for information on data export, I've created an explanation here in a community-maintained FAQ: https://hackmd.io/@patcon/conversa-calls/https%3A%2F%2Fhackmd.io%2F%40patcon%2FB1rwrXiu_#How-do-I-export-my-data ❤️