Dataverse: Unable to upload with Swift external storage

[PaulBoon]

I am unable to upload a file to a Swift store is based on openstack version 2.21.0.2.
I get an authentication failure.

[2019-06-04T13:01:27.353+0000] [glassfish 4.1] [SEVERE] [] [] [tid: _ThreadID=50 _ThreadName=Thread-9] [timeMillis: 1559653287353] [levelValue: 1000] [[

Command exception, HTTP Status code: 401 => UNAUTHORIZED
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.javaswift.joss.exception.HttpStatusToExceptionMapper.getException(HttpStatusToExceptionMapper.java:48)
at org.javaswift.joss.exception.HttpStatusExceptionUtil.getException(HttpStatusExceptionUtil.java:16)
at org.javaswift.joss.exception.HttpStatusExceptionUtil.throwException(HttpStatusExceptionUtil.java:10)
at org.javaswift.joss.command.impl.core.httpstatus.HttpStatusChecker.isOk(HttpStatusChecker.java:30)
at org.javaswift.joss.command.impl.core.httpstatus.HttpStatusChecker.verifyCode(HttpStatusChecker.java:38)
at org.javaswift.joss.command.impl.core.AbstractCommand.call(AbstractCommand.java:50)
at org.javaswift.joss.client.impl.ClientImpl.createAccount(ClientImpl.java:110)
at org.javaswift.joss.client.impl.ClientImpl.createAccount(ClientImpl.java:25)
at org.javaswift.joss.client.core.AbstractClient.authenticate(AbstractClient.java:35)
at org.javaswift.joss.client.factory.AccountFactory.createAccount(AccountFactory.java:30)
at edu.harvard.iq.dataverse.dataaccess.SwiftAccessIO.authenticateWithSwift(SwiftAccessIO.java:776)
...

[2019-06-04T13:01:27.354+0000] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=50 _ThreadName=jk-connector(3)] [timeMillis: 1559653287354] [levelValue: 900] [[
Failed to save the file, storage id swift://endpoint1:hdl-12345:FK2/IFP7D3/16b22944a50-f28ddec45274 (SwiftAccessIO: failed to authenticate keystone_v3 for the end point endpoint1)]]

I can upload using Cuberduck and curl, so I have correct credentials and connection.
Below part of my jvm-options for Dataverse (4.14)

-Ddataverse.files.swift.defaultEndpoint=endpoint1
-Ddataverse.files.swift.tenant.endpoint1=dans_dataverse
-Ddataverse.files.swift.username.endpoint1=dans_dataverse
-Ddataverse.files.swift.password.endpoint1=swiftpassword-alias
-Ddataverse.files.swift.temporaryUrlExpiryTime=3600
-Ddataverse.files.swift.endpoint.endpoint1=https://proxy.swift.surfsara.nl
-Ddataverse.files.storage-driver-id=swift
-Ddataverse.files.swift.authUrl.endpoint1=https://proxy.swift.surfsara.nl:5000/v3/auth/tokens
-Ddataverse.files.swift.authType.endpoint1=keystone_v3

I have also set the hashkey, but that one is not shown here.

Also I tried Dataverse v4.10.1 (with swift.properties file) lots of variations in the settings, and v2 and basic authentication, none of it worked.

My best guess is that this is the same problem as described in https://github.com/IQSS/dataverse/issues/5750
and the JOSS library is not sending the project json with the authentication request.

[pdurbin]

@PaulBoon thanks for opening this issue.

@carlosmcgregor do you have any ideas on this one?

[larsks]

I think @PaulBoon is correct; this sounds like the same problem I reported in #5750. The workaround is to assign the user a default project in keystone; the fix is to patch JOSS to perform authentication correctly.

[pdurbin]

Over at http://irclog.iq.harvard.edu/dataverse/2019-06-04#i_96054 @larsks suggested the patch might be in https://github.com/javaswift/joss/pull/156 (thanks!)

[PaulBoon]

To see if the 'patch' fixed my problem I built a Dataverse (based on the develop branch) that uses the new joss lib version 0.10.3.
In Dataverse SwiftAccessIO.java I then use AuthenticationMethodScope, to fix the keystone v3 request:

           account = new AccountFactory()
                .setAuthenticationMode(AuthenticationMethodScope.PROJECT_NAME)

For a real fix this 'scope' should be made a configuration/option, but I wanted to get some positive results first.
Somehow, this still did not give me a working connection, maybe it is time to try connect with another SWIFT endpoint, just to see that it can work.
For future testing it might be nice as well to have a dockerized SWIFT endpoint that we know Dataverse can work with.

[pdurbin]

@PaulBoon bummer that the fix on the joss side didn't help. 😢 Thanks for testing it.

[PaulBoon]

In order to understand what is going wrong I tried to get a look at the requests and responses that are being sent. With Wireshark I only see the encrypted content, and I was not able to get it to decrypt for me. Using mitmproxy was also problematic, because I run Dataverse on a VM (centos7 with Vagrant). Maybe I can get is to work for me if I use a proper server to install it on, but my focus is now on using S3 instead of SWIFT as external storage.

This is because I got it working with the S3 API of the openstack storage at surfsara.
For this I had to change the source code in S3AccessIO.java and add enablePayloadSigning and disableChunkedEncoding. I probably need to do a PR to make these into options.

[pdurbin]

@PaulBoon interesting. Yes, if you needed to change S3AccessIO.java a pull request would be appreciated!

[carlosmcgregor]

Hello @pdurbin ! Sorry I could not get back to you earlier. I was away for a couple of weeks and just came back.

@PaulBoon , I see that the snippet of your settings says:

-Ddataverse.files.swift.password.endpoint1=swiftpassword-alias

I was wondering which steps you took to create the password alias since in my configuration file I see:

-Ddataverse.files.swift.password.endpoint1=${ALIAS=swiftpassword-alias}

[PaulBoon]

I am sure I have done what is explained in the guide:
http://guides.dataverse.org/en/latest/installation/config.html#swift-storage

So I used the following on the commandline:

./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.files.swift.password.endpoint1='${ALIAS=swiftpassword-alias}'"
If I run this and then list the options it has:

-Ddataverse.files.swift.password.endpoint1=swiftpassword-alias

If this is indeed wrong, than that might explain my authentication problem.
To get what you have, I need to escape the $ character:

./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.files.swift.password.endpoint1='\${ALIAS=swiftpassword-alias}'"

I will check if this is indeed fixing my problem.

[PaulBoon]

I just checked it with the escaped '$' in the asadmin, and it works!

Also applied the joss fix to 4.10.1 (we have 4.10.1 in production) and configured it with the properties file (not the asadmin for the password alias); and that worked also.
This does not mean that we don't need the joss fix, because I have not had a working swift in 4.10.1 without that fix.

[carlosmcgregor]

Happy to hear it worked, @PaulBoon ! Out of curiosity, what OS are you running Dataverse on?

[carlosmcgregor]

@pdurbin , could you please assign this issue to me? I would like to update the documentation to include the missing \. My apologies for this error.

[PaulBoon]

@carlosmcgregor I am using centos7,
great that you want to fix it.
My focus is now on S3, probably I will create an issue for the payload signing etc.

[pdurbin]

could you please assign this issue to me?

@carlosmcgregor sure I just invited you to join https://github.com/orgs/IQSS/teams/dataverse-readonly/members so that I can assign this issue to you.

[carlosmcgregor]

Thank you, @pdurbin :) Accepted the invitation!

[PaulBoon]

@pdurbin just createde an issue for the extra S3 options here #5996.