K-9: Onboarding Re-work

[philipwhiuk]

I expect this ticket to be continually updated to cover the bits of onboarding we need to add. It's a meta-ticket for the Onboarding milestone to avoid opening tons of issues on an already busy repository

Future comments will cover design, planning new activities and workflows

Expected behaviour

Users should be able to easily setup accounts, identities and encryption etc.

The default case should be as simple as possible.

Actual behaviour

Current setup doesn't support lots of stuff

Things to support

• [ ] Manual configuration
• [ ] Autoconfiguration of email settings
• [ ] - Mozilla
• [ ] - Outlook
• [ ] Existing provider list
• [ ] Identity setup
• [ ] Client certificates
• [ ] Using both client certificates and password auth #793
• [ ] Google XOAuth
• [ ] Hotmail & Yahoo XOAuth
• [ ] Read only e-mail
• [ ] Certificate viewing and pinning
• [ ] User friendly certificate error handling
• [ ] - Expired
• [ ] - Self-Signed
• [ ] User friendly TLS / SSL error handling
• [ ] Encryption key set-up
• [ ] Signature
• [ ] Connection testing

[philipwhiuk]

Activities

The current flow is shown below. To start account the Basics activity is launched. It then passes control to one of two activities depending on whether we have information on the email provider.

Note that several call CheckSettings with it being responsible for checking incoming and outgoing settings. It then returns control to the activity that called it.

Currently we don't pass around much data between the activities. Instead we manipulate an Account which is created incomplete in the Accounts list.

AcceptCertificateDialog

We currently have a dialog (in CheckSettings) that is shown when a certificate isn't valid. But it needs to be reworked to use either a DialogFragment or become a full activity. Also it doesn't display certificates and errors in a friendly way

An initial mock-up for this:

[philipwhiuk]

And here's a rework of AccountCheckSettings. Note that you'd not actually see this exact view (I'm just enabling it for testing) - the spinner and text above it is what you'd see before it had attempted a connection. The later information is what you'd see afterward.

Right now ACS just goes straight through if all your details work. This turns it into more of a review screen.

I'm still in two minds about what exactly to display for the cert. Chrome shows OrganisationName, [Country] if it's an EV certificate I think. My guess is most email servers aren't EV certs because nobody shows this info for email connections. So all you're likely to get is the CommonName (something like '*.google.com'). Which is fine. I added the root CA because it seemed useful though it doesn't really tell you that much.

The previous image had the cert details in (what would be if I finished it off) a very detailed format, similar to how you get if you delve into the cert in Chrome. I still want to make that available on this screen but I'm not quite sure the UI right now.

Just to get all this information will require some sizeable modifications to the stores so that ACS can get it from a connection. I'm not quite sure how that API will work.

I don't want to do too much UI work here as it'll have to be re done for #645 but to allow some of the above requirements it does need to be concepted to an extent.

[rugk]

Should the last screenshot be the one when an invalid cert is presented?

In this case there is

1. missing a big warning that this whole stuff is insecure (and there could be any values displayed, because well… self.signed certs) and…
2. missing the display of the fingerprint or hash of the public key, which are the only way to verify that the certificate is legit

Other ideas:

• smaller "Confirm" button
• highlight "amend" (why "amend?" should not this be at least "cancel" or so?) and rename it to "back to safety" or so (no joke)

Have a look at how browsers (or Thunderbird e.g.) nowadays do it (when they are presented with self-signed certs). You should do it in a similar way.

[philipwhiuk]

No. The first screenshot is that and both are very much WIP.

missing the display of the fingerprint or hash of the public key, which are the only way to verify that the certificate is legit

None of that covers pinning. I haven't even got to it yet. It's just reviewing the connection that was tested. There'll be another but that says 'Review Certificate Chain' or something and then another screen which shows the certificate chain and all the details (similar but different to the first image which is for cert errors) with options to pin at each step of the chain.

why "amend?"

Because it goes back to the server details to allow you to amend them. There'll be a separate screen for pinning.

Have a look at how browsers (or Thunderbird e.g.) nowadays do it (when they are presented with self-signed certs). You should do it in a similar way

I agree. And there's a whole ton of stuff to implement. Right now we have very limited certificate management code.

[cketti]

I'd like a rework of the account setup experience be driven by what we want the UX to be like. Not by the current code structure.

With Let's encrypt being a thing there's no good reason for servers to use self-signed certificates. I'm counting on this becoming an even more exceptional case than it is right now.
And I don't see a reason to show any kind of certificate information in the common case. Ideally I want account setup to work automatically. The user enters an email address and a password and the app finds the correct server settings and sets up the account automatically.

Manual certificate pinning is a super-advanced topic that not many people need. I think there are more important features that should get our attention before we even think about how a decent UI for certificate pinning could look like.

[rugk]

Yeah, I think I missed the topic of this mockup. I thought it's shown for cert validation failures, but if it is purely "informational" it is of course all right.

[jancborchardt]

Ideally I want account setup to work automatically. The user enters an email address and a password and the app finds the correct server settings and sets up the account automatically.

@cketti exactly. There should only be 3 inputs:

• Display name (can be retrieved automatically from the phone I guess?)
• Email address (can also be prefilled from the phone, at least the first one? When there is an account with that address already, leave empty)
• Password (can’t be prefilled, right?)

Then the first field with no content should be automatically focused and keyboard opened. That’s the most seamless flow.

[philipwhiuk]

Password (can’t be prefilled, right?)

The first screen shouldn't even have a password. OAuth and certificate-only based auth doesn't use them.

[cketti]

Client certificates are an edge-case. I'd say the user has to manually opt-in for that (hide the option in the overflow menu).

How to support both OAuth and password-based authentication is an interesting question.

[njeyaakili]

While I would agree that the ideal default account setup might be fairly automatic, I think that there still needs to be an "advanced" option that allows for manual input of values, including certificate acceptance.

With certificates, an issue with Let's Encrypt is that it requires that at least a dns record for the "servername" be in public view. People running mail servers on intranets (these may be for internal-only mail, or for mail that is moved from public to private servers in some manner) may not feel comfortable with that -- even if the only detail that is in the public DNS for the host is a TXT record -- so may still resort to self-signed certs, so would still need a manual way to accept them.

[philipwhiuk]

An email account on a phone that only works while connected to work WiFi seems like an awfully odd edge case but we have no intention to remove any functionality here - more to optimize the steps for the default case. It's about making the 90% case a lot easier while maybe adding one or two taps for the weird cases.

[cketti]

When running mail servers on an intranet, set up your own PKI. If you tell users to manually accept certificates you should feel bad!
That being said, we won't remove the ability to shoot yourself in the foot anytime soon. We might make it slightly more difficult, though.

[ghost]

Hey all! I'm the designer that worked on the earlier K-9 redesign back in 2013. I've been talking to @cketti + @jancborchardt lately, trying to move things forward design-wise.

Design to me is meaningless without serving actual people. It was imperative to listen to what K-9 users—in the dev community or otherwise—had to say about K-9. I’m excited to have a K-9 user survey out there with over 100 responses! I’ll consolidate findings and will share more info soon. For now, feel free to participate: https://uxquestionnaire.typeform.com/to/SHacN5

Onboarding
In light of all this new data, my old K-9 design project needs some re-work. Onboarding is a great start, and an opportunity to improve the overall app experience. I’m attaching a few lo-fi screens I’ve designed as a first pass.

Please ignore the grayscale colors and placeholders:

It's all work in progress, but feel free to provide your thoughts.

One question I had was around PGP setup. Can we use Android Intent to connect the PGP portion of the flow to a third party like OpenKeychain?

In comparison to the previous attachment, here is K-9’s current flow:

And Gmail’s add account flow:

[rugk]

I am sorry, but actually I like @moctodot's design in https://github.com/k9mail/k-9/issues/1859 more. Your suggested flow looks nice from a "how it should work" perspective and is really minimalistic, but currently it looks a bit like Windows Phone/Windows 10 style, especially with the big top title on white background.
Also it is not really material design (already the top bar is missing).
But good you look at GMail. There you can see how to do it in material design properly and I think you should do it in a similar way (with such a header etc.).

[moctodot]

@juliakorbut great work! I was currently working on the onboarding experience aswell :D It's not as detailed as your concept though and the tutorial part is still missing. Once that's finished I'll add it to #1859. Unfortunately I don't have much time right now so that could take a while :(

[moctodot]

Also can you get to server settings screen without selecting the manual setup? Because when I add my account all of that seems to happen automatically.

[rugk]

Also can you get to server settings screen without selecting the manual setup? Because when I add my account all of that seems to happen automatically.

I think this happens when K9mail does not recognize the mail provider you use. If you use such a big one as gmail you of course should not see this screen.

[philipwhiuk]

Yeah - so right now we have a manually maintained list of configurations. We want to augment this with automatic configuration.

Ideally manual configuration should be rarely needed but there's always going to be some use for it (and right now we re use those screens for editing server settings).

[moctodot]

@philipwhiuk @rugk good to know! Thanks for clearing that up.

[ghost]

@rugk that's fair. At this stage this is mostly a wireframe. Larger typography helps simplify the flow and make it appear shorter, even when there are multiple screens such as those I had here. Material design doesn't enforce an app bar in all times.

@moctodot I like how you made the icon face-on, it simplifies the shape. Contextually though, I'd say this flow isn't really the place to configure account color. It's not helping the current task at hand and the user doesn't even know what that means at this stage.

Also, tutorials or overlays like that aren't really effective since the majority of users cancel them right away. It's better to make the interface easier to use in the first place.

I'm working on the rest of the flow right now, and consolidating findings from the user survey. It'll give everyone insight into what's important to users, and where K-9 is sorely lacking. There are a lot of things beyond the core interface, icon, or visuals, that need to be designed.

It would be great if user stories were created for these, and we could tackle the design tasks together. Do you use Sketch? Whats the best way to reach you via email?

[ghost]

My question still stands re: PGP. Can a user tap on 'Setup PGP' and complete the action via Android Intent to OpenKeychain and the likes?

[rugk]

I really like the idea to configure the account colour there. What it means? It's a colour assigned to the account - that's all the user needs to know (and does know) in this step.
Alternatively you could just assign a random color and let the user change it later.

And I really like the idea of having a tutorial (just showing the most basic things, such as swiping right/left to delete/archive/… a mail or so). That's good to explain and even if a user cancels it, they are able to trigger it in the settings - as shown in the mockup.

[moctodot]

@juliakorbut you're probably right about the account color. The best solution would probably still be to choose a random color initially to change it later from the settings. Instead of a complete tutorial it could be an option to display tips contextually from time to time, like the guidelines suggest it. I don't know how much help I can be since I don't have much time at the moment. I use Inkscape for my designs, simply because I can't afford anything else :D

[ghost]

That's totally fine! I'm trying to find a way in which we can work together, not necessarily synchronously. I want to share some of the user survey results with you so we can create user stories and work in parallel. How can I contact you offline? You can email me through here.

[Valodim]

@juliakorbut Since I'm a core contributor to both K-9 and OpenKeychain, development is very close. What we can do is only limited by required effort and technical reason, if we can come up with a concept that works in terms of these two, we can do whatever we want! :+1:

Regarding OpenPGP setup, this also involves the autocrypt project where a couple of people (including me) are working on a draft spec towards a coherent user experience that takes both secret and public key management out of the user's hands.

Generally, there are only three cases we have to care about:
1) The user doesn't have OpenKeychain installed. We can think about doing onboarding, but right now that's a Thing For Later™
2) The user has no secret key for a given email address, and wants to create one. We actually need zero user input for this, since key creation depends on nothing save the email address itself.
3) A user already has a secret key, and needs to import it. As long as we don't have anything in terms of a sync mechanism (and we won't for a long time), this requires the user to export the secret key from somewhere else and import it locally. This breaks the flow in such a harsh way that I think we shouldn't do it in the setup wizard but just tell the user where they can do it later.

@juliakorbut @moctodot Do either of you happen to be at the Internet Freedom Festival in Valencia, three weeks from now?

[ghost]

Thanks for your reply, @Valodim !

Very interesting challenge. The user survey I've sent out had people repeatedly mention the need for reliable integration with OpenKeychain.

Is it possible to solve a lot of the import/export problems on the OpenKeychain side? For instance, you set up a new email account in K-9, and when you're done setting the basics up you app-switch to OpenKeychain and manage/export/import the keys from there?

Not going to the IFF festival—have K-9 contributors presented there in the past?

[ghost]

Another recurring request in the survey is for S/MIME support.

Would it be a similar import/management process?
Enabling POP for all mail > creating an account certificate > importing that certificate > adding that certificate so you can sign with it on an account > sending signed emails to another contact as a 'handshake'

[philipwhiuk]

Yes, essentially the implementation and management for S/MIME is expected to be similar - a third-party app like OpenKeychain (or perhaps a future release of OpenKeychain) that handles trusting certificates etc. There's an issue for it #1003 but developing the new app / adding the functionality to OpenKeychain is quite a big project in of itself.

[philipwhiuk]

Rework of the ugly as hell accept certificate dialog (see #2355 and a prior prototype above) into a couple of activities. Still prototyped (these are just Android Studio layouts).

I'm semi-hopeful that after my imminent holiday I'll be able to find time to actually get this version into the code base. The dialog is unhelpful and very horrible so it really needs changing.

[rugk]

:+1:

But please don't highlight the 'accept cert'. Although (or better: because it is) red users will tap on it immediately.
Instead hide it as much as possible and require at least two taps before the user can select it.

(Maybe move it into the "view cert info" field)

And highlight the "reject cert" button (with green or so)

[ghost]

@philipwhiuk How do both screens interact? Is the first one an expanded view of the second one?

Also, where do you envision it living in relation to this flow, from the design repo? https://raw.githubusercontent.com/k9mail/k-9-design/master/Flows/K-9%20-%20Entire%20App.png

That info could be helpful for me to design something.

[philipwhiuk]

@juliakorbut

So right now the onboard process is more like:

1, 2, 4, *, 5

The * screen is a 'Validating settings' screen. You don't really see it if the process is pretty quick. Similarly if it's really quick to check the auto-config stuff 3 won't be seen for long either.

In that scheme my screens sit between * and 5. When you click Accept/Reject it would relaunch * and if it was all fine then continue to 5.

[ghost]

@philipwhiuk The redesign has a simple setup process, that after it's done displays various additional steps (that are optional and/or advanced). Could cert handling be part of those additional steps?

https://raw.githubusercontent.com/k9mail/k-9-design/master/Flows/K-9%20-%20Entire%20App.png

[ghost]

Also, would you mind updating to this icon? https://github.com/k9mail/k-9-design/tree/master/App%20Icon

[rugk]

Also, would you mind updating to this icon?

I liked the ones proposed in https://github.com/k9mail/k-9/issues/1723 better.

Your general app design looks good too, but it should be merged with the ones proposed in https://github.com/k9mail/k-9/issues/1859.

[ghost]

@rugk You're welcome to have your own preferences and use whichever icon you like.

The icon I linked here was part of the K-9 design repo, a thorough redesign project I started in 2013. It already includes user testing, high fidelity design for all screens, color themes, icon (in all sizes and optically corrected to fit a variety of devices and wallpaper colors), etc. It's also part of the K-9 org.

[ArchangeGabriel]

For anyone stumbling upon here about the new icon, the discussion is now in https://github.com/k9mail/k-9-design/issues/1.

[philipwhiuk]

Back on the original topic:

@juliakorbut

The problem is that if the certificate or connection details are invalid the account just won't work at all. We really have to be checking we can actually connect to both the incoming and outgoing settings before we properly create the account. And it's during that connection that we may see the certificate error.

If we add the pinning feature, that can be done as a post step like PGP. But invalid certificates are not optional.

I'll try and draft a more complete flow with progress arrows. It might be a bit crude because I don't have Photoshop/good design skills :(

[ghost]

That's totally fine! I think I understand it better now due to your explanation. So, if I get it correctly, it's a step that _might_ appear, but only if issues arise? In some cases, the cert handling is automatic?

[philipwhiuk]

Yep. Normally the certification is fine so we won't show it at all.

[ghost]

So in an instance where we do get to the cert screen, what are the possible outcomes?

Fail, succeed, anything else? If it fails, does the user have to start all over again? Can they change specific attributes in the cert until it does work?

Also what do you mean by pinning?

[Valodim]

If a cert fails, it's a problem on the server side, which fails to correctly identify itself with an identity that is deemed trustworthy by the user's Android OS. There is not much that the user can do other than complain to their provider, however this scenario is exceedingly uncommon with professionally operated providers in the first place.

Not sure if this has been mentioned before, but am invalid cert here is precisely the same thing happening like in your browser when it tells you "this website is not safe! (make exception?)"

One way out of this is a manual override to "just trust" the cert, which is as convenient as it is dangerous. The connection will work, but won't be authenticated so could be eavesdropped upon. "pinning" means that this invalid cert will be accepted, but only this one, so the server can't give a different (invalid) one next time.

There - fortunately - are no parameters to fiddle around with on the client side that could make the cert valid after all.

[philipwhiuk]

The pinning feature is mostly for valid certificates. It prevents attacks where rogue certificate authorities issue certificates incorrectly.

Acceptance is like pinning but for invalid ones.

[Valodim]

Pinning of valid certificates is an expert feature, I'm fairly sure that we can disregard it in account setup.

One thing we might want to have here is proxy support, if that ever gets done. In particular, if Orbot is installed, we could ask if the user wishes to use that for connection. But that's not something we need to worry about now.

[rhari991]

For people working on this as part of GSoC 2017, should the new UI be material (as shown above), or should it match the current theme ?

[philipwhiuk]

It'll depend on the theme of the app at the time. I'd expect the author to work with the K-9 Design team to co-ordinate it. Account Setup is somewhat independent of the rest of the app so that may help as well.

Certainly there will be a fairly sizeable UI portion to this one, whichever way it goes.

[ghost]

@philipwhiuk since this issue was created by you, do you want to sketch out a very simple flow for the cert process? Even simple scanned drawing on paper. I can then take that, design based on it, and discuss here further.

I saw the screens you posted here but I just want to make sure I'm getting the big picture correctly - the sequential order of things, and small details I might be missing.