Generator-jhipster: Migrate logstash appender from UDP to TCP

[anjia0532]

_LoggingConfiguration.java#L92-L96

        // Limit the maximum length of the forwarded stacktrace so that it won't exceed the 8KB UDP limit of logstash
        ShortenedThrowableConverter throwableConverter = new ShortenedThrowableConverter();
        throwableConverter.setMaxLength(7500);
        throwableConverter.setRootCauseFirst(true);
        logstashAppender.setThrowableConverter(throwableConverter);

the logstash-plugins-inputs-udp.html#plugins-inputs-udp-buffer_size

buffer_size
Value type is number
Default value is 65536
The maximum packet size to read from the network

65536/1024=64K

logstash/issues/2111#domenkozar

Yes, I spend a few hours on chasing a bug. I've fixed the bug on the client side to split messages, but then logstash was parsing JSON in a weird way if it hit the 8k limit.

the logstash input json like this

{
    "_index": "logstash-2017.07.07",
    "_type": "jhipster",
    "_id": "AV0d78jVDn-FIhloNkzA",
    "_version": 1,
    "_score": null,
    "_source": {
        "@timestamp": "2017-07-07T16:44:00.078Z",
        "instance_name": "%{app_name}-10.42.196.67:%{app_port}",
        "@version": "1",
        "host": "10.42.196.67",
        "message": "{\"@timestamp\":\"2017-07-08T00:44:00.063+08:00\",\"@version\":1,\"message\":\"....{long string}.....

[anjia0532]

Log statements with stack trace get truncated due to Logstash 8 KB limit#generator-jhipster/3364

i remove this code and output the 100Kb msg,but the logstash only preserve 64kb(tune up buffer_size or receive_buffer_bytes is invalid)

        // Limit the maximum length of the forwarded stacktrace so that it won't exceed the 8KB UDP limit of logstash
        ShortenedThrowableConverter throwableConverter = new ShortenedThrowableConverter();
        throwableConverter.setMaxLength(7500);
        throwableConverter.setRootCauseFirst(true);
        logstashAppender.setThrowableConverter(throwableConverter);

why not use logstash-logback-encoder#tcp-appenders ?

[gmarziou]

Maybe the 8KB limit does not exist anymore or maybe it depends on your environment.

[anjia0532]

@gmarziou thanks for your reply. the logstash5.x default buffer_size it's 65536 (64KB), 2.x buffer_size it's 8K

[anjia0532]

the throwableConverter.setMaxLength(7500) it's 7500 char in java 1 char=2byte, so 7500char=15000b=14.64K > 8K

[gmarziou]

OK, I'm still using Logstash 2 but current jhipster-console uses Logstash 5 since Oct 2016 so we should change this configuration.

Now about UDP vs TCP, I don't think this was discussed before first implementation, so I guess it could be changed in code or made configurable in application.yml.

Would you like to propose a PR if other agree too?

[PierreBesson]

@anjia Thanks for the feedback on this. @gmarziou and I have been the only one to work on the integration with logstash (With some help from @cbornet if I remember right). But we have not followed closely the development of logstash and the logback logstash appender.

If you think the 8K limit has changed please submit a PR. As for using the TCP appender rather than the UDP appender I don't know what to think. I initially chose UDP for concerns about performance and it works well. What would be the reasons to use TCP ?

As an aside, I'm not sure chars are always 2 bytes I think they are 1 byte most of the time (if you are on utf8) but you are right to point out that 1byte != 1 char.

[jdubois]

Be careful of UDP packet size - I wouldn't send big messages here

[PierreBesson]

@jdubois Well then maybe that's the reason to move to TCP !

[jdubois]

I believe UDP is much better for logs: it's faster, and if lose a packet it isn't very important

[jdubois]

Interestingly, I originally copied our conf from on what Papertrail does, and it does recommend TCP (probably because of encryption):
https://github.com/papertrail/logback-syslog4j

They also have a recommendation on UDP packet size:
https://github.com/papertrail/remote_syslog/issues/68

[anjia0532]

@gmarziou maxLength's javadoc

Limits the total length in characters of the trace

It is invalid for large massage.

ShortenedThrowableConverter.java#L280-L283

        if (builder.length() > maxLength) {
            builder.setLength(maxLength - ELLIPSIS.length());
            builder.append(ELLIPSIS);
        }

[anjia0532]

@PierreBesson stackoverflow#Isn't the size of character in Java 2 bytes?

@jdubois @PierreBesson

Internally, the TCP appenders are asynchronous (using the LMAX Disruptor RingBuffer). All the encoding and TCP communication is delegated to a single writer thread. There is no need to wrap the TCP appenders with another asynchronous appender (such as AsyncAppender or LoggingEventAsyncDisruptorAppender).

The TCP appenders will never block the logging thread. If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.

The TCP appenders will automatically reconnect if the connection breaks. However, events may be lost before Java's socket realizes the connection has broken.
from logstash-logback-encoder#tcp-appenders

tcp appenders support

Multiple Destinations
SSL

i'll continue testing tcp appenders and submit a PR

[jdubois]

OK, don't hesitate to challenge also our Logback configuration (not only for Logstash):

• This originally is my own code for my own projects, and I was very careful about having a good configuration
• But that work was done probably more than 5 years ago, so many things might have changed !!

[anjia0532]

@jdubois i submit tcp appender PR and demo https://github.com/anjia0532/tcpappender, you can test it

@PierreBesson the logstash input plugin must be change to tcp input plugin,the PR https://github.com/jhipster/jhipster-console/pull/69

[PierreBesson]

Okay, so let's go along with this change. However this move from UDP to TCP is a breaking change for the Console, so we will need to release a v3 and warn users. This is a bit of work to test everything but I'm confident that using TCP is a good idea, even if only to be able to use encryption. Also both papertrail and the logstash appender seem to recommend TCP over UDP for log forwarding.
I will try to do it this week. The goal being that everything must work out of the box like before.

[gmarziou]

@PierreBesson, do you think we could enable the 2 protocols on different ports in JHipster Console? It would enable a transparent migration for existing services.

[PierreBesson]

@gmarziou, I don't see why not ! Looking at the do s seems to be supported (https://www.elastic.co/guide/en/logstash/5.4/configuration-file-structure.html). I will try it.

[anjia0532]

@gmarziou +1

i angree with enable the two ports in the JHipster Console.

the input plugin like this

# env JHIPSTER_UDP_PORT 5000
# env JHIPSTER_TCP_PORT 5001
input {
    udp {
        port => "${JHIPSTER_UDP_PORT}"
        type => syslog
        codec => json
    }
    tcp {
        port => "${JHIPSTER_TCP_PORT}"
        type => syslog
        codec => json_lines
    }
#....
}

@PierreBesson this is my demo https://github.com/anjia0532/tcpappender/blob/master/README.md#start-application

curl http://localhost:8080/api/tcp?str=jHipster&size=50000&padStr=

#str:the String to pad out, may be null
#size:the size to pad to
#padStr:the String to pad with, null or empty treated as param `str`

//...
    @GetMapping("/tcp")
    public String getActiveProfiles(@RequestParam String str,@RequestParam int size,@RequestParam(required=false) String padStr) {
        String leftPadStr= StringUtils.leftPad(str, size, StringUtils.defaultIfBlank(padStr, str));
        log.info(leftPadStr);
        return MessageFormat.format("str:{0},pad size:{1,number},pad str:{2},result str:{3}", str,size,StringUtils.defaultString(padStr, str),leftPadStr);
    }
//...

[PierreBesson]

@anjia0532, we can use the same port for both UDP et TCP, it doesn't cause any issue for the server :smile:. So let's keep using port 5000.

[anjia0532]

@PierreBesson yes, you r right. we can have both UDP and TCP requests on same port ,because each request is identified by a quintuple contained by source IP ,Destination IP, Source Port, Destination Port, PROTOCOL(as protocol can be TCP or UDP).