Gitea: Feature: Allow interacting with tickets via email

[MTecknology]

As far as I can tell, gitea has no support for receiving email responses from users. When a user receives a message, their only option for response is to open the web interface and use a web browser. This is rarely convenient for discussions while mobile, especially if your git services passwords are >200 chars. I believe this is a critical feature that needs discussion and implementation.

From my perspective, the big picture pieces are:
1a. fetch messages
1b. receive messages

1. validate messages (including duplicate check)
2. append information to the ticket

(Note: ticket -> issue or pull)

1a. - I think IMAP is the standard. Gitea is already has source that interacts with IMAP/S services. It could just be a matter of marking a message as read, pulling it down, and dropping it into a processing queue. Of course, offering a DELETE_MESSAGE=(true|false) will always be appreciated.

1b. - Listening for an SMTP connection sounds neat, but please don't ever consider it. This would be a pretty scary attack surface for quite a number of reasons. (I typed out a lot here, but... tl;dr-- postfix kicks butt at being a mail server, gitea should not reinvent that wheel). Instead, I'd like to suggest listening on a unix socket.

In settings where this would be observed, you can expect to see custom-written middleware that does things like check the spamassassin score. This middleware then opens a socket to write the message content to.

In the case of 1b, it's almost exactly like 1a, except that it listens to a socket for messages instead of connecting to an outside service, trying to handle duplicate message race conditions, etc. The former (1a) would always work well for single-node setups where the latter (1b) would be available for handling scale.

2. - I envision another process pulling messages off of a queue (another redis db sounds nice) and doing some validation before letting the message be included in a ticket.

I'd like to start with opinions (stated as expectations):

• A user may only respond to messages; always reply, never compose.
• A user may only respond to the ticket the message was sent from,
• Any computational verification should be minimally invasive (not taxing).
• Verification should be nearly stateless; avoid storing large tables of tokens.
• A user may not use their response to impersonate another user. (@strk: fun times!)
• User-visible tokens must be cryptographically secure.

Now, for the fun part... I've been poking around at how github handles this and I think would could handle a simple implementation that meets (my) expectations. This would require adding a "salt" column to the user table.

I have a larger vision of an overall workflow, but starting the the explanation at this point is easiest. At 2. we will have a From: address that looks similar to github's. This is in the form "reply+\@reply.domain.tld", where \ is a series of hashed and non-hashed values. We will also have an In-Reply-To: field that tells us where this message should be directed.

With \ taken out, we can dissect it into it's various components (my thoughts):

• $user-id => Our copy of the user-id, hex value, left zero-padded to 10 digits
• $key => A cryptographically pseudo-random (i.e. cat /dev/urandom | md5sum) value the user gets
• $cksum => A value we generated with a combination of the $key + \

From the Message-ID field, we will have these components available:

• $locator => \/\/\/\
• $replied-to => The specific message being replied to. (or just the timestamp of it)

[ note: ^ the sources of these values are what I skipped earlier .. coming up soon ]

To verify this message is a reply from a message we sent to that user (email.. considered good enough for password resets), we'll be able to run the formula:

user_hash = checksum($user-id + $key + _user_salt($user-id) + $locator + $replied-to)
if $user_hash != $cksum { drop_message() }
elif get_post_date($replied-to) >= 90days { drop_message() }
elif # could possibly also check From: and verify it came from an address on file (maybe...)
else { post_message($locator, $user-id, $message-body) }

That works because... When we sent the message, we set two fields: Reply-To and Message-ID.

Ex: Message-ID: <go-gitea/gitea/issue/2351/[email protected]> ; the exact structure could be different, but I think this is fairly straight-forward (note: the produced message-id [https://en.wikipedia.org/wiki/Message-ID])must always) be globally unique--except when sending the same message to multiple recipents). This is the message the user is replying to and it's post-date gets to be used in our TTL/validity calculation. The mail client should turn this into In-Reply-To for us to verify later.

Ex: Reply-To: reply+<glob>@domain.tld ; This is where we'll want the message routed. We should be able to customize the exact "reply" user. This allows formats such as mycorp-inbound+<glob>@gmail.com without causing routing problems because this is a format that gmail permits.

The \ portion was explained earlier. It takes the form: $user-id$key$cksum.

2. summary:

In the end, all this does is generate a random value, combine it with a user salt and some other things, and then append some extra information to the email.

When we get the message back we verify that we can re-produce the $cksum they provided. Because they've never had access to the salt, this is a value they can't produce on their own. Provided the key is sufficiently large, I believe this is more than adequate to provide this feature without opening the doors for abuse. This allows us to automatically invalidate old tokens with one simple check (is $message-id older than X days?).

= Discussion!

I know this is massive, and probably not without mistakes, but I'd prefer get the discussion started since I can't see one person implementing this by themselves.

[lunny]

I like do that via discord webhooks. Like the mechanism we developing gitea. Every day I received github and drone notifications from discord app on iOS or macOS. And then I click the link to open the browser to visit it. We also could filter the notification such as commit, issues, pull requests to different channels if the notification are more and more.

[MTecknology]

@lunny Wouldn't discord webhooks be something entirely unrelated to this? I don't think I understand.

[strk]

Great work Micheal !

This would require adding a "salt" column be added to the user table.

There is one already :)

In the end, all this does is generate a random value

What's the random value for ?

[MTecknology]

In the end, all this does is generate a random value

Every single message sent out gets a random value. The random value is combined
with the user's salt (and other bits) to generate the checksum value. The
theory being that you can't manipulate any values (such as user-in or
in-reply-to) without invalidating the checksum value.

What's the random value for ?

You have part (the random part) of the one-time passphrase used to generate the
checksum that verifies the rest of the values, but it's useless without the
salt.

You'd have to look at the source code to grab the assembly/verification bits
and start trying to brute force the salt value. heh.. if we included a custom
per-app security token in the hash generation, that'd make brute-forcing a
user's salt exponentially more challenging (providing a second unknown). At
least I think so?

If this doesn't make sense, I'll try again after sleep. :)
(a whiteboard image might be helpful to make sure this is explained thoroughly...)

[strk]

What's the random value for ?

You have part (the random part) of the one-time passphrase used to generate the
checksum that verifies the rest of the values, but it's useless without the
salt.

But then you need to store the random value to a table, or you cannot
verify the checksum ?

[strk]

I guess the title of this ticket could be made clearer, like:

Allow interacting with tickets via email

[MTecknology]

@strk and I were discussing the random part a bit further and, while needed, there's no need for the user to ever see it. We can produce a predictable pseudo-random key using a collection of known values. This seems a much better idea.

So then the flow (my attempt at representing the flow using only text):

• User posts a comment [comment-id:48] to an issue [uri:/go-gitea/gitea/issue/2386]
• The comment was posted 24 Mar 2017 @ 16:05 CST [$post-time:1503695121]
• Two users are subscribed: user-id = [204, ,208]
• Values that already exist in gitea: $secret-key, user[$user-id]['salt']
• Each user will receive the same message, but each message will have a slightly different header
• Each message will have the same Message-ID: ${uri}/${comment-id}.${post-time}@${fqdn}
• Each message will have a unique Reply-To: <reply-user>+<glob>@<fqdn> header
• The body of the message will be exactly the same thing we already send now

The reply-user and fqdn need to be customizable because that's the portion that makes up the destination where gitea will find the message. The \--or "label" as gmail likes to call it--is just along for the ride.

The \ is built by:

challenge = cksum( ${message-id} + ${user-id} + _get_user_salt(${user-id}) + $secret-key) )
reply-to = ${reply-to-user}+${challenge}${user-id}@${reply-to-fqdn}

==>
Message-ID: go-gitea/gitea/issue/2386/[email protected]
Reply-To: r[e-mail]

The first 56 chars are the cksum value (I used sha224 for that one) and the remaining digits up to the '@' are the user ID.

When the user responds to that message, their mail client will add the header In-Reply-To: ${message-id}. It will also set a To: and From: header (basic.. I know).

When we receive the reply, we can parse the following values (or else drop the message):

- $from-addr
- $challenge, $user-id = '^.+?\+(?<challenge>[0-9a-f]{56})(?<userid>[0-9]+?)\@.*$' << $to-header
- $uri, $comment-id, $post-time = '^.*\s(?<uri>.+)\/(?<commentid>[0-9]+?)\.(?<posttime>[0-9]+)\@.*$' << $message-id

To validate the message:

$from-email, $challenge, $user-id, $uri, $comment-id, $post-time = _parse_headers()
if _get_time() > ${post-time} + _config_key('REPLY_VALIDITY_TTL'): drop_message()
if not _user_exists( ${user-id} ): drop_message()
if ${from-addr} not in ${user-email-list}: drop_message()
if not ${challenge} = cksum( ${message-id} + ${user-id} + _get_user_salt(${user-id}) + $secret-key) ): drop_message()
post_message()

[strk]

it looks like it takes a database query to verify every incoming mail, how about only using $secret-key (per-server key) ?

[MTecknology]

Maybe it's just a feel-good thing, but having a little bit of per-user entropy tossed in sounds like a good idea in my head. The user-id by itself might be sufficient, but it seems weak to me. (I can't argue with knowledge, just my gut.)

[jonasfranz]

Could we use jwt for this, since we could store the user id, issue id in it. Responses coming in will be verified without database queries, just checking the signature. In addition we can limit what the replier of the email may do.

[Superpat]

I'd love to see work on this issue continue. I often have to explain to people that they cant respond to the emails sent by gitea.

[dereks]

Several Issue Trackers / Ticket Trackers support this feature, for example, the Open Source Eventum Issue Tracker (PHP). It is very useful when interacting with a public customer base, so the users can just "reply" to the email even if they don't know how to use a web browser. (Think grandparents...) Mailing List software does this also.

I've seen two systems used for email input into a ticket/mailing list/discussion thread.

The first method is based on the email's Reply-To address that the users reply to. The software creates a magic Reply-To address for every outbound message so that any replies get processed correctly. Mailing List software like Majordomo and Mailman do this. Majordomo requires a specialized Postfix configuration where Postfix parses the incoming email addresses, looks for special formatting such as [email protected] and then Postfix directs it to the software's local user account for processing. Mailman does it by dynamically updating Postfix's aliases file. This is a pain to set up because you must run your own MX server, but it allows you to tie into Unix's server-side Inbox processing filters like .filter or Sieve. These days I would recommend against this.

The second method is (as suggested above) is much easier, just using an IMAP Inbox and sending with a Subject: that has a magic cookie in it. It requires creating an Inbox account on some other email server someplace. It could even be a gmail.com address. Then the software periodically checks it with polling, or has a dedicated thread that waits for IMAP NOTIFY messages so polling isn't needed. This is the easiest to set up because there is no need to run your own MX server. One of my VPS providers uses a ticket tracker with this method, and here is what their emails look like:

[Ticket ID: 113718] My Ticket Subject Here

The [Ticket ID: 113718] is what gets parsed, and 113718 is the magic cookie. That could be modified to be something like [REPO_NAME (token)] Topic Name From Issue Tracker Here. This is useful because any email client that lets you "reply" generally keeps the Subject line in tact (plus an extra "Re:" prefix).

Finally, it's possible to include the token in the email body (like at the end in the signature). This requires any Reply-To to keep the email contents (it can't be just a blank email that retains only the Subject line, so it is easier for users to delete the token by accident).

I would advise using the Subject: line for the token because it's the most user-friendly and least error-prone. A JWT would be good cryptographically but would be too long to put into the Subject: line. I recommend just a simple 6-char string like "d26280". It is safe enough for posting a reply to an Issue.

Another feature would be having emails to "[email protected]" automatically result in the creation of a new Issue on a repository, where the Subject is the title of the issue. Maybe the same IMAP inbox could be used, but if there is no [REPO_NAME (token)] prefix then it treats it as a new Issue.

I am looking forward to this feature as it would make Gitea more suitable for a I.T. Support organization (using more of the Issue Tracker and Wiki features than the git version control). However most of Gitea's target audience knows how to use a web browser, and even GitHub doesn't offer Reply email handling.

[westurner]

Is there a good go library for SMTP / POP / IMAP?

Is the Gitea process the correct place for such a server?

Would it be more advisable to integrate with an existing mail processing
server? What authentication should be used for this?

What are the events?

• new inbound email without issue token
• new inbound email with issue token
• issue title modification
• issue desc modification
• issue new comment
• new issue
• new PR
• new commit

Does it make sense to have a more general notification service that
supports e.g. Web Notifications, App Notifications, and email? Getting the
page or app to update without waiting for a full page refresh is basically
the same problem?

https://www.w3.org/TR/push-api/

https://notifications.spec.whatwg.org/

On Wed, Aug 12, 2020, 1:11 AM Derek Simkowiak [e-mail]
wrote:

Several Issue Trackers / Ticket Trackers support this feature, for
example, the Open Source Eventum Issue Tracker (PHP). It is very useful
when interacting with a public customer base, so the users can just "reply"
to the email even if they don't know how to use a web browser. (Think
grandparents...) Mailing List software does this also.

I've seen two systems used for email input into a ticket/mailing
list/discussion thread.

The first method is based on the email's Reply-To address that the users
reply to. The software creates a magic Reply-To address for every outbound
message so that any replies get processed correctly. Mailing List software
like Majordomo and Mailman do this. Majordomo requires a specialized
Postfix configuration where Postfix parses the incoming email addresses,
looks for special formatting such as [email protected] and then
Postfix directs it to the software's local user account for processing.
Mailman does it by dynamically updating Postfix's aliases file. This is a
pain to set up because you must run your own MX server, but it allows you
to tie into Unix's server-side Inbox processing filters like .filter or
Sieve. These days I would recommend against this.

The second method is (as suggested above) is much easier, just using an
IMAP Inbox and sending with a Subject: that has a magic cookie in it. It
requires creating an Inbox account on some other email server someplace. It
could even be a gmail.com address. Then the software periodically checks
it with polling, or has a dedicated thread that waits for IMAP NOTIFY
messages so polling isn't needed. This is the easiest to set up because
there is no need to run your own MX server. One of my VPS providers uses a
ticket tracker with this method, and here is what their emails look like:

[Ticket ID: 113718] My Ticket Subject Here

The [Ticket ID: 113718] is what gets parsed, and 113718 is the magic
cookie. That could be modified to be something like [REPO_NAME (token)]
Topic Name From Issue Tracker Here. This is useful because any email
client that lets you "reply" generally keeps the Subject line in tact (plus
an extra "Re:" prefix).

Finally, it's possible to include the token in the email body (like at the
end in the signature). This requires any Reply-To to keep the email
contents (it can't be just a blank email that retains only the Subject
line, so it is easier for users to delete the token by accident).

I would advise using the Subject: line for the token because it's the most
user-friendly and least error-prone. A JWT would be good cryptographically
but would be too long to put into the Subject: line. I recommend just a
simple 6-char string like "d26280". It is safe enough for posting a reply
to an Issue.

Another feature would be having emails to "[email protected]"
automatically result in the creation of a new Issue on a repository, where
the Subject is the title of the issue. Maybe the same IMAP inbox could be
used, but if there is no [REPO_NAME (token)] prefix then it treats it as a
new Issue.

I am looking forward to this feature as it would make Gitea more suitable
for a I.T. Support organization (using more of the Issue Tracker and Wiki
features than the git version control). However most of Gitea's target
audience knows how to use a web browser, and even GitHub doesn't offer
Reply email handling.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/go-gitea/gitea/issues/2386#issuecomment-672578408,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAMNS6OH3DWYFUHYWBE7W3SAIP7HANCNFSM4DYIQFSA
.

[ptman]

https://godoc.org/net/smtp but not really for receiving. The best solution usually is to let postfix handle smtp and have an entry in aliases that passes the message to a program for processing. Something like:

issues: "|/usr/local/bin/gitea-process-incoming"

[westurner]

Apprise supports a bunch of notification services and also SMTP/SMTPS:
https://github.com/caronc/apprise#email-support

Gorush and gotify don't appear to specifically support SMTP or SMTPS.

• [ ] is there already a (persistent configuration pub/sub) event system in the codebase?
• [ ] apprise for outbound notifications?
• [ ] a set of functions for processing inbound email with initial adapter(s) for [postfix,]?

[westurner]

(I think that sending some or most emails through apprise would be advantageous because additional notification services could be used for: updating the displayed page when there's a new comment by pushing a websocket event, pushing some notifications to whatever channel)

[westurner]

Edit: Forgot that apprise is written in Python. Looks like shoutrrr https://github.com/containrrr/shoutrrr is written in go, inspired by apprise, and also supports gotify https://github.com/gotify/server

[dereks]

@ptman I recommend against forcing users to run an SMTP (Mail Exchange) server. Users would need to own a domain, set up their own DNS MX records, and deal with Spam attacks.

Instead, why not just let Gitea be an IMAP client that points to an Inbox somewhere? Then users can just point to any email server, including GMail.

https://github.com/emersion/go-imap

IMAP includes an "IDLE" extension which allows push-based email notifications. The IMAP client library should take care of that. As far as updating the HTML, that would be a feature independent of any email tie-ins. The HTML refreshes should just as if a new comment was posted with a browser, and not be different for an email.