I am using NEST to retrieve filebeat logs from ELK. I can see logs in Kibana with @timestamp but when i retrieve documents with NEST timestamp is null
This is object to retrieve documents:
public class FileBeatDto
{
public DateTime timestamp { get; set; }
public string datetime { get; set; }
public DateTime received_at { get; set; }
public int offset { get; set; }
public string version { get; set; }
public string input_type { get; set; }
public int count { get; set; }
public Beat beat { get; set; }
public string host { get; set; }
public string source { get; set; }
public string message { get; set; }
public string type { get; set; }
public object fields { get; set; }
public List<string> tags { get; set; }
}
This is the query:
var result = elk.Search<FileBeatDto>(x => x
.Index("filebeat-*")
.AllIndices()
.Type(type)
.Query(q =>
q.Match(qs => qs.Field("fields.asset_tag").Query(asset_tag)) &&
q.Match(qs => qs.Field("message").Query(filter))
)
.Take(lines)
);
NEST/Elasticsearch.Net version: 2.4.3
Elasticsearch version: 5.0-alpha5
Description of the problem including expected versus actual behavior:
@timestamp not retrieved by NEST
This is most likely because it's @timestamp in the index, but only timestamp is inferred based on the POCO property name.
A few ways to deal with this...
Use InferMappingFor<> on your ConnectionSettings to "rename" the property:
new ConnectionSettings()
.InferMappingFor<FileBeatDto>(m => m
.Rename(f => f.timestamp, "@timestamp")
)
or use the Date or JsonProperty attributes:
[Date(Name="@timestamp")]
public DateTime timestamp { get; set; }
[JsonProperty("@timestamp")]
public DateTime timestamp { get; set; }
This is likely the same issue as #2207.
Also, you're using NEST 2.4.3 which technically is not compatible with ES 5.0.0-alpha5. See the compatibility matrix. You should jump on the latest alpha release of NEST instead.
thanks! adding the attribute to POCO works and can retrieve timestamp now.
Closing this and #2207.
Most helpful comment
This is most likely because it's
@timestampin the index, but onlytimestampis inferred based on the POCO property name.A few ways to deal with this...
Use
InferMappingFor<>on yourConnectionSettingsto "rename" the property:or use the
DateorJsonPropertyattributes:This is likely the same issue as #2207.
Also, you're using NEST 2.4.3 which technically is not compatible with ES 5.0.0-alpha5. See the compatibility matrix. You should jump on the latest alpha release of NEST instead.