Conductor: HTTP TASK comes out with socketTimeoutException

[rajanigk]

When HTTP Task is using the uri localhost there is no issue. When it uses any uri which is not localhost and does a post, it fails with the error "response": "com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out". The same task with same workflow works with conductor code which is about 2 months old. Also with the latest code if i comment the following lines

client.setReadTimeout(defaultReadTimeout); |   client.setConnectTimeout(defaultConnectTimeout);

From the file contribs/src/main/java/com/netflix/conductor/contribs/http/RestClientManager.java

it works.

I have used the following taskdefnition :

{
"name": "test_http_post",
"description": "test http post",
"createTime": 0,
"createdBy": "String",
"retryCount": 1,
"timeoutSeconds": 3600,
"timeoutPolicy": "RETRY",
"retryLogic": "FIXED",
"retryDelaySeconds": 60,
"responseTimeoutSeconds": 3600,
"concurrentExecLimit": 0,
"inputTemplate": {},
"rateLimitPerFrequency": 0,
"rateLimitFrequencyInSeconds": 1
}
And following workflow

{
"name": "test_http_post",
"taskReferenceName": "test_http_post",
"type": "HTTP",
"startDelay": 0,
"inputParameters":{

             "http_request": {
                     "uri": "https://abcd.regionname.amazonaws.com/testworkflow",
                     "method": "POST",
                     "contentType":"application/json",
                     "body":"${workflow.input}"
             }
    }
   }

IS anything wrong with my definitions? Or is it a defect in conductor code?

[kishorebanala]

@rajanigk This is not related to Conductor and a typical Http request error. The task was not able to connect to provided url. The timeouts I believe are reasonable at around 5 secs. Try checking other connection issues from curl or similar tools to debug.

[rajanigk]

After debugging the conductor code, i was able to figure out how to set the connection and read timeout for httptask. In the workflow they need to be set as input parameters.
{
"name": "task1",
"taskReferenceName": "task1",
"type": "HTTP",
"startDelay": 0,
"inputParameters":{

             "http_request": {
                     "uri": "<URL>",
                     "method": "POST",
                     "contentType":"application/json",
                     "body":"${workflow.input}",
                     "connectionTimeOut":"3600",
                     "readTimeOut":"3600"
             }
    }
   }

[tburschka]

Damn, (why) is it (not) documented? I tried this for a whole day...

[rajanigk]

Where is it documented? Did i miss seeing the documentation?

[rituparnopal]

It's not working for me at all. Even though I set this value Conductor seems to read only 150 seconds default read time allotted for HTTP tasks.

[merlin-northern]

hullo.

looking at the code
contribs/src/main/java/com/netflix/conductor/contribs/http/RestClientManager.java:34-35:

    static final String HTTP_TASK_READ_TIMEOUT = "http.task.read.timeout";
    static final String HTTP_TASK_CONNECT_TIMEOUT = "http.task.connect.timeout";

and testing this, what does work is puting in the conductor configuration following lines to set the timeouts (all in milliseconds):

    http.task.read.timeout=15000
    http.task.connect.timeout=15000

btw the default values of 150ms and 100ms are not always realistic for the real-life applications (apart from localhost, as the original post stated).

cheers,
peter

[rituparnopal]

hullo.

looking at the code
contribs/src/main/java/com/netflix/conductor/contribs/http/RestClientManager.java:34-35:

  static final String HTTP_TASK_READ_TIMEOUT = "http.task.read.timeout";
  static final String HTTP_TASK_CONNECT_TIMEOUT = "http.task.connect.timeout";

and testing this, what does work is puting in the conductor configuration following lines to set the timeouts (all in milliseconds):

    http.task.read.timeout=15000
    http.task.connect.timeout=15000

btw the default values of 150ms and 100ms are not always realistic for the real-life applications (apart from localhost, as the original post stated).

cheers,
peter

Yep, worked for me when I realized its in milli-seconds.

[merlin-northern]

in case anyone is reading that, this patch fixes the apparent problem with deserialisation of tasks.inputParameters.http_request.connectionTimeOut and readTimeOut on some setups:

diff --git a/contribs/src/main/java/com/netflix/conductor/contribs/http/HttpTask.java b/contribs/src/main/java/com/netflix/conductor/contribs/http/HttpTask.java
index f5485718..b40d02a6 100644
--- a/contribs/src/main/java/com/netflix/conductor/contribs/http/HttpTask.java
+++ b/contribs/src/main/java/com/netflix/conductor/contribs/http/HttpTask.java
@@ -152,11 +152,11 @@ public class HttpTask extends WorkflowSystemTask {
    protected HttpResponse httpCall(Input input) throws Exception {
        Client client = rcm.getClient(input);

-       if(input.connectionTimeOut != null ) {
+       if(input.connectionTimeOut>0) {
            client.setConnectTimeout(input.connectionTimeOut);
        }

-       if(input.readTimeOut != null ) {
+       if(input.readTimeOut>0) {
            client.setReadTimeout(input.readTimeOut);
        }
        if(input.oauthConsumerKey != null) {
@@ -307,9 +307,9 @@ public class HttpTask extends WorkflowSystemTask {

        private String oauthConsumerSecret;

-       private  Integer connectionTimeOut;
+       private int connectionTimeOut;

-       private Integer  readTimeOut;
+       private int readTimeOut;



@@ -453,22 +453,22 @@ public class HttpTask extends WorkflowSystemTask {
        /**
         * @return the connectionTimeOut
         */
-       public Integer getConnectionTimeOut() {
+       public int getConnectionTimeOut() {
            return connectionTimeOut;
        }

        /**
         * @return the readTimeOut
         */
-       public Integer getReadTimeOut() {
+       public int getReadTimeOut() {
            return readTimeOut;
        }

-       public void setConnectionTimeOut(Integer connectionTimeOut) {
+       public void setConnectionTimeOut(int connectionTimeOut) {
            this.connectionTimeOut = connectionTimeOut;
        }

-       public void setReadTimeOut(Integer readTimeOut) {
+       public void setReadTimeOut(int readTimeOut) {
            this.readTimeOut = readTimeOut;
        }

without it the readTimeOut and connectionTimeOut are null and the timeouts just seem not to be working.

[kishorebanala]

@merlin-northern Wondering if you have an example where Integer object works, while primitive int fails.

[merlin-northern]

@merlin-northern Wondering if you have an example where Integer object works, while primitive int fails.

thanks for commenting. I have the opposite, my case is: with the Integer type the timeouts in input do not work at all. I have actually connected jdb and saw null Integer fields in the input. With the above patch it works. I think this is what @rituparnopal saw as well. would you consider using int types in the Input class?

the setup where I have Integer fields in Input class not being deserialised:

# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (IcedTea 3.12.0) (Alpine 8.212.04-r0)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)
Alpine Linux 3.9
Kernel: 4.14.97-90.72.amzn2.x86_64
# ldd `which java`
    /lib/ld-musl-x86_64.so.1 (0x7fcb6b540000)
    libjli.so => /usr/lib/jvm/java-1.8-openjdk/lib/amd64/jli//libjli.so (0x7fcb6b52a000)
    libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fcb6b540000)
    libz.so.1 => /lib/libz.so.1 (0x7fcb6b510000)

conductor v2.11.0 and tested also the latest.

[rituparnopal]

@merlin-northern Wondering if you have an example where Integer object works, while primitive int fails.

thanks for commenting. I have the opposite, my case is: with the Integer type the timeouts in input do not work at all. I have actually connected jdb and saw null Integer fields in the input. With the above patch it works. I think this is what @rituparnopal saw as well. would you consider using int types in the Input class?

the setup where I have Integer fields in Input class not being deserialised:

# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (IcedTea 3.12.0) (Alpine 8.212.04-r0)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)
Alpine Linux 3.9
Kernel: 4.14.97-90.72.amzn2.x86_64
# ldd `which java`
  /lib/ld-musl-x86_64.so.1 (0x7fcb6b540000)
  libjli.so => /usr/lib/jvm/java-1.8-openjdk/lib/amd64/jli//libjli.so (0x7fcb6b52a000)
  libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fcb6b540000)
  libz.so.1 => /lib/libz.so.1 (0x7fcb6b510000)

conductor v2.11.0 and tested also the latest.

For my case it was an issue of milliseconds. The default value of 100 ms and 150 ms was too low for my microservices.