Dataverse: Glassfish 5 Upgrade

[djbrooke]

Spun out of #4217. We are more confident with a Glassfish upgrade than we are with Java EE8/Java 9 upgrades. We'll plan to take Glassfish to conclusion and leave the spike as a separate issue.

[djbrooke]

@pameyer - please feel free to drop notes of the issue re: login that you experienced.

[pameyer]

quick check of this branch w\ glassfish5 + apache. curl on top-level url worked as expected (redirect to login page); using a browser triggered a glassfish exception (and failed to allow login).

[2017-10-31T15:13:58.494+0000] [glassfish 5.0] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=50 _ThreadName=jk-connector(2)] [timeMillis: 1509462838494] [levelValue: 900] [[ GRIZZLY0013: Exception during FilterChain execution java.lang.NullPointerException at org.glassfish.grizzly.http.ajp.AjpHandlerFilter.encodeHttpPacket(AjpHandlerFilter.java:282) at org.glassfish.grizzly.http.ajp.AjpHandlerFilter.handleWrite(AjpHandlerFilter.java:245) at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1092) at org.glassfish.grizzly.http.io.OutputBuffer.flushBinaryBuffers(OutputBuffer.java:1066) at org.glassfish.grizzly.http.io.OutputBuffer.flushAllBuffers(OutputBuffer.java:1036) at org.glassfish.grizzly.http.io.OutputBuffer.close(OutputBuffer.java:754) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:244) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:179) at java.nio.channels.Channels$WritableByteChannelImpl.implCloseChannel(Channels.java:469) at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115) at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:365) at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:178) at org.primefaces.application.resource.PrimeResourceHandler.handleResourceRequest(PrimeResourceHandler.java:87) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:667) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1580) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:338) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:305) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:250) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:250) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:652) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:591) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:463) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:168) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:242) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) at java.lang.Thread.run(Thread.java:748) ]]

[matthew-a-dunlap]

I have spent about a day on this and made little progress aside from gaining background knowledge and identifying some errors.

Works: Login & accountRegistration

Broken:

1. Dataverse cannot be published, as the DvObject id is being set to null by some process
2. Search only works with a fix to stop some process from setting the active page to 0. Issue seems similar to the one above. Error attached from when setPage() is removed to show error trace.
3. Dataverse cannot be created, a warning is thrown in the logs and the returned page renders all hidden sections in a broken way.
4. Dataset cannot be created, ControlledVocabularyValueConverter errors out because injected DatasetFieldServiceBean is null.

1. Dataverse cannot be published.txt
2. Search starts with page == 0.txt
3. Dataverse cannot be created.txt
4. Dataset cannot be created.txt

[pdurbin]

Huh, this is interesting, from 2.Search.starts.with.page.0.txt above: javax.el.PropertyNotWritableException: The class 'edu.harvard.iq.dataverse.search.SearchIncludeFragment' does not have a writable property 'page'. at javax.el.BeanELResolver.setValue(BeanELResolver.java:433) . My related commit: 6f72baa

I don't know what to say about the other errors. I was able to create dataverses and datasets via API on Glassfish 5. I didn't test JSF much.

[matthew-a-dunlap]

@pdurbin That log message is the one from when I removed the getPage() setter. Was an easy way to get a useful stack trace (the trace when it errored on page == 0 hid a lot).

[matthew-a-dunlap]

From standup, we decided to let this wait until technical open hours next Tuesday for discussion.

[djbrooke]

We revisited this in backlog grooming 11/15:

• For this to be considered done, we need to have the application running on Glassfish 5, have Vagrant and Docker Images up to date, and have the installer run Glassfish 5.
• If it's easy to support both Glassfish 4 and Glassfish 5, we should do so, but the primary goal is to get to Glassfish 5.

[poikilotherm]

Hey guys,

I am working on a deployment of Dataverse at Forschungszentrum Jülich in Germany. This might be a good solution to get a research data repository in place and we want to compare Dataverse with B2SHARE.

As @pdurbin mentioned in some other issue/thread, I pretty much dislike patching Glassfish 4.1, too. Any chances this issue or PR #4244 hits a release anytime soon? I am willing to help in testing. I'm neither a JEE expert nor a Glassfish expert, but I know a bit or two about Java web dev (Spring) and I'm willing to dig deeper... ;-)

Thx,
Oliver

[pdurbin]

@poikilotherm hi! Thanks for being willing to test! The best way to get a sense of where issues are in the queue is to look at the kanban board at https://waffle.io/IQSS/dataverse

As of this writing this issue is in the "Backlog" column with a couple dozen of issues in front of it. This is a better spot to be than the "Inbox" column where new issues land when they are created. We are constantly shuffling the order of issues in the backlog.

As I indicated at https://github.com/IQSS/dataverse/issues/4172#issuecomment-340774938 I've been concerned that Glassfish isn't updated very often compared to Payara. It sounds like your main concern is the patching you have to do and yes, I wrote about this at #2628.

[poikilotherm]

Hi @pdurbin,

I will be on vacation for the next three weeks. After that I will come back to this topic...

Cheers
Oliver

[pdurbin]

I've been meaning to mention that I was recently alarmed to read the following in https://www.eclipse.org/org/foundation/boardminutes/2019_03_26_Minutes.pdf

"Mike noted that the future of Eclipse Glassfish beyond version 5.2 is uncertain due to resource constraints."

[djbrooke]

Closing in favor of #6230