Documentation: Intermittent `java.lang.IllegalStateException: Connection pool shut down` on clean install

[dannylamb]

While testing the fixes required for https://github.com/Islandora-CLAW/CLAW/issues/686, I ran into this every 10 or so times I set up Karaf with Api-X and Alpaca.

2017-08-15 11:55:43,966 | INFO  | pool-52-thread-3 | InitMgr                          | 124 - fcrepo-api-x-jena - 0.3.0.SNAPSHOT | Caught exception while initializing.  Trying again in 1000 ms
java.lang.IllegalStateException: Connection pool shut down
    at org.apache.http.util.Asserts.check(Asserts.java:34)[54:org.apache.httpcomponents.httpcore:4.4.5]
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:185)[54:org.apache.httpcomponents.httpcore:4.4.5]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:257)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.fcrepo.apix.registry.HttpClientFetcher$CloseableHttpClientProxy.doExecute(HttpClientFetcher.java:115)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)[53:org.apache.httpcomponents.httpclient:4.5.3]
    at org.fcrepo.apix.jena.impl.LdpContainerRegistry.exists(LdpContainerRegistry.java:280)
    at org.fcrepo.apix.jena.impl.LdpContainerRegistry.lambda$init$8(LdpContainerRegistry.java:131)
    at org.fcrepo.apix.jena.impl.InitMgr$1.lambda$$11(InitMgr.java:70)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

A quick googling leads me to believe that there's timing issues where Api-X needs to close connections, and that's shutting down the connection pool.

https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html#setConnectionManagerShared(boolean) seems to be a solution according to stack overflow https://stackoverflow.com/a/28646605.

@ajs6f Thoughts? I know we talked about whether or not we consider our custom http cllient to be shared when you were doing the initial work.

[ajs6f]

Hm. My first (really rough) thought is that we can triple-check that all CloseableHttpResponses are really getting closed (try-withresource FTW) and then that we might want to adjust any conneciton pooling. I'd really rather not be fooling around with manually closing connections or pools and we (in theory) shouldn't have to.

The HttpClient itself is definitely thread-safe and shareable-- that's part of its basic contract. Now, if we are mucking about with its underlying componentry (e.g. connection pools) we have to support that guarantee by doing what we are doing in a thread-safe shareable way...

[dannylamb]

I'll file an issue with Api-X and see if we can sort this out.

[DiegoPino]

PoolingHttpClientConnectionManager...

https://stackoverflow.com/questions/25889925/apache-poolinghttpclientconnectionmanager-throwing-illegal-state-exception ?

[ajs6f]

In API-X mvn dependency:tree | grep httpcomponents gives 4.4.5 and 4.5.2, so not the bug. Possibly the code needs to be updated, which should probably come under the ticket @dannylamb filed. Otherwise, @DiegoPino are you thinking that the problem is in CLAW? Because that stacktrace is clearly from API-X.

[DiegoPino]

@ajs6f yeah, the bug is prior 4.4, I see some settings and option to set the shared pool and i think it is something in API-X, but we(claw) are triggering it. So it is not our error, and agree with the sharing part
```JAVA
.setConnectionManagerShared(true)
````

[ajs6f]

Cool, I will continue tracing through.

[DiegoPino]

@ajs6f is it https://github.com/fcrepo4-labs/fcrepo-api-x/blob/63022f47ec32e05e689e138220eee56aef713561/fcrepo-api-x-registry/src/main/java/org/fcrepo/apix/registry/HttpClientFactory.java#L117-L118? Can we tune the config there when we deploy CLAW + the shared client part?

(I´m on a meeting so distracted... sorry)

[ajs6f]

@DiegoPino This is our injected client. I think we might need to adjust there.

[dannylamb]

@DiegoPino Those were my initial thoughts too after reading the stack overflow page I posted in the issue description.

[DiegoPino]

@dannylamb i missed your link, jajajaja. Same one i pasted...feel so silly :man_facepalming: so funny.

[dannylamb]

@DiegoPino We're just on the same page, man. Literally. =)