Sendgrid-csharp: scopes.get() results in error 'utf8' is not a supported encoding name

[andrewchilcott]

Issue Summary

Whenever I call sg.client.scopes.get() I receive an error message - 'utf8' is not a supported encoding name.

Any reference to utf8 should actually refer to utf-8. Is there a typo in the source code?

Steps to Reproduce

This is the code that I am using:

        public static async Task<ScopeList> getScopes()
        {
            ScopeList result = new ScopeList();
            dynamic response = await Shared.sg.client.scopes.get();
            if (response.StatusCode.Equals(HttpStatusCode.OK))
            {
                result = JsonConvert.DeserializeObject<ScopeList>(response.Body.ReadAsStringAsync().Result);
                result.userMessage = "Success - " + result.Scopes.Count.ToString() + "  retrieved at " + DateTime.Now.ToString() + ".";
            }
            else
            {
                result.userMessage = Shared.GenerateFailureMessage(response, "scopes_get_all", "");
            }
            return result;
        }

    public class ScopeList
    {
        [JsonProperty(PropertyName = "scopes")]
        public List<string> Scopes { get; set; }
        [JsonIgnore]
        public string userMessage { get; set; }
    }

The error gets thrown from Response.Body.ReadAsStringAsync().Result (a method that I use successfully on all other calls to the api)

There is valid content in the response.Body object. The buffer contains 2486 characters and I have decoded the first bit which shows { "scopes": [ alerts.create , alerts...... etc - I am assuming that the rest of the response is valid.

Technical details:

• sendgrid-csharp Version: 8.0.5
• .NET Version: 4.5.2

[Jericho]

@andrewchilcott I faced the same problem when I was adding a method in the StrongGrid library to retrieve the "scopes" for the current user.

I confirm that a charset is included in the response from the /scopes endpoint. This is not a problem per se, but it is different than all other SendGrid endpoints which omit the charset.

I also confirm that the charset specified in the response is invalid. utf8 is not a valid value, it should be utf-8 (notice the missing dash). For reference, please see this site where all acceptable charsets are defined and you will be able to confirm utf-8 is the correct spelling.

Finally, I confirm that Microsoft's HttpClient chokes when you invoke ReadAsStringAsync because it doesn't know which encoding to use to convert the body of the response into a string. To be clear: Microsoft's HttpClient chokes because it tries to automatically convert the binary content in the response into a string but this conversion fails due to the presence of the invalid charset.

Here's the workaround that I implemented in StrongGrid, hopefully this will work for you as well until SendGrid solves the issue on their end: as I said, the issue is when you attempt to read the body of the response as a string so the solution is to read it as a stream and manually convert the content of the stream into a string using the encoding of your choice (in this case, the right encoding is Encoding.UTF8)

var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);
var responseContent = string.Empty;
using (var sr = new StreamReader(responseStream, Encoding.UTF8))
{
    responseContent = await sr.ReadToEndAsync().ConfigureAwait(false);
}

@thinkingserious I have contacted SendGrid support and opened a ticket (#806220). I was told that a feature request would be created to remove the charset from the /scopes response and therefore ensure the endpoint's behavior is consistent with all other endpoints.

[andrewchilcott]

Thank you so much for your effort

Andrew

[thinkingserious]

@andrewchilcott,

We have added this to our backlog for a fix. Thanks for taking the time to report this to us!

@Jericho,

Thanks for jumping in with a great solution, once again :)

[Jericho]

I received a follow-up from the support engineer handling the case I opened regarding this issue:

Indeed, IANA only mentions about UTF-8 which is correct, but in some programming languages, you can use the syntax utf8 instead of utf-8. So it is only about the syntax, not about the actual charset type

I checked with our senior engineers and it seems that the API call only returns that response at the moment, and cannot be changed. That is because the call returns data that is specifically UTF-8 encoded, and needs to be present in the call's response headers.

Unfortunately, based on this statement it doesn't seem like SendGrid will be correcting this issue.

[andrewchilcott]

So it appears that utf is correct in some languages but not in c# and Sendgrid won’t change their code to work in all languages!

Thanks for the work around, its working fine so we can now leave it until some other poor c# developer pulls their hair out trying to get it to work and opens yet another support ticket!

Andrew

[thinkingserious]

Thanks for following up @Jericho!

[thinkingserious]

@andrewchilcott,

I'm leaving this ticket open so that I can add @Jericho's fix to this library.

[Jericho]

I improved the workaround I suggested earlier and converted it to an extension method for convenience. When you invoke this extension method you can either specify the encoding you desire or you can allow it to automatically determine the appropriate encoding. In cases where the encoding cannot be determined (due to the charset being omitted or invalid), Encoding.UTF8 is used by default.

// The null parameter will cause the encoding to be determined automatically
// If the encoding cannot be determined, Encoding.UTF8 is used as the fallback
var responseContent = await response.Content.ReadAsStringAsync(null).ConfigureAwait(false);

You can find the source for this extension method here:
https://gist.github.com/Jericho/fa5867eb93c4df63ed59ee66f5f468db

[tony-ho]

I can take this one.

I propose a more direct workaround than those @Jericho suggested:

• check the charset in the response in SendGridClient.MakeRequest()
• if the value is the invalid utf8 value, update the response to the correct value, utf-8

The advantage of this approach is the HttpContent will not be have to be read as stream to convert to a UTF-8 string, then again when users call response.Body.ReadAsStringAsync() (as in the examples).

This also removes the need for the improved ReadAsStringAsync() extension method.

[opengtd]

@tony-ho Is there still anything that needs to be done here? Looks like a fix is pending in [PR 786] (https://github.com/sendgrid/sendgrid-csharp/pull/786 )

[tony-ho]

@opengtd #786 resolves this issue, all that needs to be done is for it to be merged.

[childish-sambino]

Fixed by #786