Pinpoint: Can we build a Pinpoint Collector running in a cluster with load balancer

[dinesh4747]

HI All -

Do we have any documentation/best practices on how to build/setup collector running in a clustered mode across different boxes and point to any of the load balancer like NGINX or Apache HttpD

[koo-taejin]

Hi dinesh4747

We are not using nginx or apache.
So, we don't have experience you are talking about.

It will be a great help if you build and share the experience.

thanks

[dinesh4747]

HI @koo-taejin

I have just mentioned NGINX/Httpd for an example

Do you aware of any solutions which is already built for collector running in clustered on LB

[dinesh4747]

Also i see in groups that people commented saying that we can build it using DNS Mutiple IP Address and L4

can you give me some context on that as it would help us in giving direction on building up the new solution

[koo-taejin]

Currently, we are getting help from network solution team, and it is difficult to say that details are confidential.
It would be better if you find and apply the commonly used methods.

It will be a great help if you build and share the experience.

[dinesh4747]

HI @koo-taejin

Sure we are trying to build up using NGINX/KeepaliveD

I shall let you know on the updates/progress

[koo-taejin]

@dinesh4747 thanks :)

[SKingNight]

@dinesh4747
hi , is it succeed building up cluster by using nginx or lvs .
In large distributed systems, one collector will not be accepted in leadership decision solution.

[SKingNight]

nginx 1.9+ stream can be used to tcp/udp load balance

[dinesh4747]

HI @SKingNight

I am trying out with GCP LB I shall post you the updates

However like whether nginx 1.9+ stream does support TCP+UCP But does it scales well it routing agent traffic to collector without any connection drop outs

Have you ever tested it out in any of the env?

[SKingNight]

@dinesh4747
Yes, I Has tried it，in nginx.org，the doc says version1.9+ support tcp/udp。But i used nginx with version 1.11.13 which installed by yum。 [root@localhost nginx]# nginx -V
nginx version: nginx/1.11.13
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) `

And the access.log
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.29 54822 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
172.19.3.36 29389 - [26/Apr/2017:11:02:53 +0800] 200 0 UDP 172.19.3.117 9996
... ...

And its nginx.conf

user nginx;
worker_processes 2;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 10240;
}

stream {
proxy_protocol_timeout 120s;
log_format main '$remote_addr $remote_port - [$time_local] '
'$status $bytes_sent $protocol $server_addr $server_port'
'$proxy_protocol_addr $proxy_protocol_port';

access_log  /var/log/nginx/access.log  main;


upstream tcpcol {
   # hash $remote_addr consistent;

    server 172.19.3.117:19994     max_fails=5 fail_timeout=30s;
    server 172.19.3.117:29994     max_fails=3 fail_timeout=30s;

}

upstream udpcol995 {
   server 172.19.3.117:19995     max_fails=5 fail_timeout=30s;
   server 172.19.3.117:29995     max_fails=3 fail_timeout=30s;
}

upstream udpcol996 {
   server 172.19.3.117:19996     max_fails=5 fail_timeout=30s;
   server 172.19.3.117:29996     max_fails=3 fail_timeout=30s;
}

server {
    listen 9994 so_keepalive=30m::10;
    proxy_connect_timeout 5s;
    #proxy_timeout 3s;
    proxy_pass tcpcol;
}

server {
    listen 9995 udp;
    #proxy_responses 1;
    proxy_timeout 20s;
    proxy_pass udpcol995;
}

server {
    listen 9996 udp;
    #proxy_responses 1;
    proxy_timeout 20s;
    proxy_pass udpcol996;
}

#server {
#    listen [::1]:12345;
#    proxy_pass unix:/tmp/stream.socket;
#}

}

[SKingNight]

@dinesh4747
in this case we can find that 9994 is tcp, 9995 and 9996 are udp.
I did not test whether the udp msg will be lost. i think lost/dorp some msg can be accepted in pinpoint.
@koo-taejin

[eysdo]

@SKingNight
Thanks,i will try it.