Schema-registry: Avro logical datatype question

[margitbomholt]

We are running Confluent platform 3.2, using Kafka jdbc connect to sink to Postgresql. And the source-program is created using kafka-rest in node.js

The following schema-part is registered in the schema registry:
{
'name': 'Test_Topic_TimeStamp',
'type': 'record',
'fields': [
{ 'name': 'Id', 'type': 'string' },
{
'name': 'TimeStampField',
'type': 'long',
'logicalType': 'timestamp-millis'
}]
}

We were expecting the created schema in the database should contain column with type "timestamp" but we get "bigint". Is it possible to get the column created as "timestamp"?

We have a similar problem with the logical type : "date"

What to do with datatype "Guid"?, we don't find that in the Avro 1.8 description

We have tried with:

{
'name': 'Test_Topic_Date',
'type': 'record',
'fields': [
{ 'name': 'Id', 'type': 'string' },
{
'name': 'DateField', 'type': 'int',
'logicalType': 'date'
}]
}
{
'name': 'Test_Topic_Guid',
'type': 'record',
'fields': [
{ 'name': 'Id', 'type': 'string' },
{
'name': 'GuidField',
'type' : 'string',
'size' : 38,
'logicalType': 'uuid'
}]
}

Any input is highly appreciated.

[ewencp]

@margitbomholt There are actually a number of translations being performed here, during which some information could potentially be lost:

1. In REST Proxy, the Avro serializers are used to convert your data to Avro. At the moment I think this is where the data is initially being lost because currently Confluent Platform ships with Avro 1.7.7. The timestamp types only appeared in 1.8 (1.7 only had the decimal type).
2. In Connect, first the data is deserialized from Avro and converted to the generic Connect Data API. This API does include logical types, but the conversion would be limited to those in the Avro version (i.e. currently decimal) as long as the source of the Avro is not generated by Connect. (Connect itself can pass extra metadata through that would be independent of the version of Avro.)
3. In Connect, the data is then passed to the connector which needs to translate it to an appropriate type for the database. The JDBC sink does translate decimal, dates, and times, and timestamps to the appropriate types, so I wouldn't expect this step to be an issue.

For UUIDs, if there isn't support in Avro, the only option is to use an encoding (e.g. a string as you have) and document the format. Connect also has not added a UUID logical type, although it is possible to pass the information through to connectors via a Connect schema name if you want to set that at the source of the data. To do so you would set 'connect.name' as a property for the field in the Avro schema. However, note that only Connectors that are aware of that logical type will handle it correctly -- most would just pass the string along as a string.

See also #434 where there is discussion about upgrading to a newer version of Avro.

[margitbomholt]

Do you know if there are any plans to ship the Connect platform with Avro 1.8 ? or is a manual upgrade possible?
We have testede the date format using JDBC Connect Source and Sink, and the datetime (timestamp) is making the Connect Source to throw an error.
Any input is highly appreciated.

[ewencp]

@margitbomholt I think we'll be bumping it to a 1.8 version in the next minor release of Confluent Platform.

[simplesteph]

@ewencp that's great to hear. As a suggestion, it'll be great to provide examples that show how to use logical types (decimals, timestamps) with that release, to encourage best practices for users. The main confusion for me comes with the kafka connect specific types (connect.something), and how these should be leveraged or not for logical types

[katsharp]

Definitely a +1 for me on better docs for this. I eventually figured out what was going wrong from reading the code.

We have the same situation as @margitbomholt.

[margitbomholt]

I'm new to the Confluent platform. How often is made a minor release of the Confluent Platform?

[ewencp]

@margitbomholt Major/minor releases are every ~4 months currently. Bug fix releases are done on demand as bug fixes are needed, but there will commonly be at least one within a month or two of a major/minor release.

[margitbomholt]

Is this issue (#522 and #434) solved in the new version 3.2.1 ?

[ewencp]

@margitbomholt We wouldn't upgrade a major version of Avro in a bugfix release, but the fix for #434 is merged in #563 and will be released with 3.3.0. So yes, this is effectively resolved now since the only remaining part of the original question was around GUIDs which Avro doesn't have a logical type for yet.