Spring-cloud-aws: SimpleStorageResource not resolved in spring-boot 2.1.x

[JurrianFahner]

In the documentation this snippet can be found:

public class SimpleResourceLoadingBean {

    @Autowired
    private ResourceLoader resourceLoader;

    public void writeResource() throws IOException {
        Resource resource = this.resourceLoader.getResource("s3://myBucket/rootFile.log");
        WritableResource writableResource = (WritableResource) resource;
        try (OutputStream outputStream = writableResource.getOutputStream()) {
            outputStream.write("test".getBytes());
        }
    }
}

This works in Finchley.RC2 as expected. But when I run the same code with only spring cloud updated to Finchley.RELEASE, I get the following class cast exception: org.springframework.web.context.support.ServletContextResource cannot be cast to org.springframework.core.io.WritableResource

In my debugger I see that the Resource should be an instance of a SimpleStorageResource, instead it is an instance of a ServletContextResource.

That is caused by the resourceLoader which shoud be a org.springframework.cloud.aws.core.io.s3.PathMatchingSimpleStorageResourcePatternResolver, instead it is an instance of a AnnotationConfigServletWebServerApplicationContext.

I use spring-boot 2.0.3

[aemruli]

Thanks @JurrianFahner for reporting. This is unfortunately a regression as we have utilized some new Spring features for loading resources (to solve some other bugs). It seems that this does not work with the Web Application context. I will add test cases with a web application context and see how to solve it.

In the meantime you could inject a org.springframework.cloud.aws.core.io.s3.SimpleStorageProtocolResolver and use that directly (pass in the location and the application context to the resolve method)

[JurrianFahner]

Well that's not so easy. Because SimpleStorageProtocolResolver needs an S3 client with credentials.

So I use RC2 in the meantime until this is fixed. My project is in dev phase, so I have some time. I imagine that it will be fixed soon.

[sebastianreloaded]

This makes the S3 resource loader indeed unusable. RC2 does work though.

Is there a plan to fix this with 2.0.x ?

[sebastianreloaded]

Is this related to #384 ?

[JurrianFahner]

Is this related to #384 ?

Nope, because I didn't use devtools. So removing devtools wouldn't fix the issue.

[denis111]

I have used this inline workaround for now

Resource resource = resourceLoader.getResource(location);

// workaround for https://github.com/spring-cloud/spring-cloud-aws/issues/348
// TODO: wait the bug to be solved and remove the "if" below
if (!resource.getClass().getName().endsWith("SimpleStorageResource") && resourceLoader instanceof DefaultResourceLoader) {
  for (ProtocolResolver protocolResolver : ((DefaultResourceLoader) resourceLoader)
      .getProtocolResolvers()) {
    if (protocolResolver instanceof SimpleStorageProtocolResolver) {
      resource = protocolResolver.resolve(location, resourceLoader);
      break;
    }
  }
}

[JurrianFahner]

@denis111 It's a nice workaround, but it's fixing the outcome, but not the root of the problem. @aemruli already mentioned that the root of the problem is due to the regression in spring cloud aws. So it should be fixed in 2.x.x branches...

[darioseidl]

@denis111 Thanks! I just started using Spring Cloud AWS and this was very helpful to me.

In my project, the same workaround looks like this (in Kotlin):

    /**
     * Override the [DefaultResourceLoader] to force resolving resources that start with `s3://` using [SimpleStorageProtocolResolver].
     *
     * Workaround for https://github.com/spring-cloud/spring-cloud-aws/issues/348.
     */
    @Bean
    @Primary
    fun resourceLoader(defaultResourceLoader: DefaultResourceLoader): ResourceLoader =
        object : ResourceLoader {
            override fun getClassLoader(): ClassLoader? =
                defaultResourceLoader.classLoader

            override fun getResource(location: String): Resource {
                return if (location.startsWith(S3_PROTOCOL_PREFIX))
                    defaultResourceLoader.protocolResolvers.findFirstInstance<SimpleStorageProtocolResolver>()!!.resolve(location, this)!!
                else
                    defaultResourceLoader.getResource(location)
            }
        }

However, while this works for a ResourceLoader, I also have this problem in a ResourceResolver,
when using a s3:// path as static resource location (ResourceHandlerRegistry.addResourceLocations). In that case I also get a ServletContextResource instead of a SimpleStorageResource as location parameter in my PathResourceResolver.

My workaround looks like this

/**
 * A [PathResourceResolver] that is able to resolve resources from S3.
 */
open class S3ResourceResolver(
    val resourceLoader: ResourceLoader
) : PathResourceResolver() {

    private val log = KotlinLogging.logger {}

    override fun getResource(resourcePath: String, location: Resource): Resource? =
        // Workaround for https://github.com/spring-cloud/spring-cloud-aws/issues/348
        if (location is ServletContextResource && location.path.startsWith("/$S3_PROTOCOL_PREFIX")) {
            val s3location = location.path.substringAfter("/")

            log.debug { "Resolving resource $resourcePath under $s3location" }
            val resource = resourceLoader.getResource(s3location.appendPath(resourcePath))
            if (resource.exists())
                resource
            else
                null
        }
        else {
            log.debug { "Resolving resource $resourcePath under $location" }
            super.getResource(resourcePath, location)
        }
}

but I believe this is caused by the same bug, and should be fixed as well.

[cjdudleybw]

Hi, I've also encountered this issue, but the workaround suggested by @denis111 does not work (same error), and doesn't really address the issue. Any other approaches that have worked, or any news on when this may be fixed?

[darioseidl]

@cjdudleybw Did you register a SimpleStorageProtocolResolver? The workaround will only work if your ResourceResolver has a SimpleStorageProtocolResolver. You can register it with the Spring Cloud AWS Spring Boot auto-configuration, or manually by declaring a SimpleStorageProtocolResolverConfigurer bean, or with configurableApplicationContext.addProtocolResolver.

[JurrianFahner]

I can confirm that in Greenwich.RELEASE the problem is fixed, and SimpleStorageResource is working.

But that's only true for spring-boot 2.1.x

[derandreasberger]

Hello,
seeing the same issue - unfortunately quite a lot of spring dependencies/not sure when what got updated (but no devtools) - Anyway: I tried to update to a newer version:

... <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.3.RELEASE</version>
    </parent>
...
+
...
 <spring-cloud.version>Greenwich.SR1</spring-cloud.version>
...

Since this was reopened - can we assume that Greenwich.SR1 is not/not completely resolving this issue?
@denis111 the workaround works, very much appreciated, thanks!
Regards,
Andy

[denis111]

Works fine for me with Spring Boot 2.1.x...

[annagapuz]

I am experiencing the same issue of the s3 protocol not resolving and getting a FileNotFoundException. Using Spring Boot starter 2.1.3-RELEASE and Spring Cloud Greenwich.RELEASE.

Will try workaround by @denis111 and post back if it works.

[annagapuz]

Update # 2:

Got it working with a different workaround.

Still on Greenwich.RELEASE and Boot 2.1.3.RELEASE.

Despite the s3: protocol, my resource is a ServletContextResource and no protocol resolvers are found in DefaultResourceLoader.

When I try to wire in SimpleStorageProtocolResolver, I get the following:
_

No qualifying bean of type 'org.springframework.cloud.aws.core.io.s3.SimpleStorageProtocolResolver' available
_

I have to manually create SimpleStorageProtocolResolver by auto-wiring in AmazonS3 and passing it in to the constructor:

SimpleStorageProtocolResolver simpleStorageProtocolResolver = new SimpleStorageProtocolResolver(amazonS3);

AWS Context XML:
<aws-context:context-resource-loader/>

POM excerpt:

<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.3.RELEASE</version>
        <relativePath/>
    </parent>

<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
        </dependency>

         <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-aws-context</artifactId>
        </dependency>

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>Greenwich.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

Hopefully a fix will be found soon!

[gysel]

I also have this issue with Spring Boot 2.1.5 and Greenwich.

@amgapuz do you have a more detailed example on how to configure the SimpleStorageProtocolResolver? I'm not sure how to do it without XML.

[darioseidl]

I have upgraded to Spring Boot 2.1.5 and Spring Cloud Context 2.1.1 and I found that this is working now if I remove Spring Boot Devtools from the classpath.

This might explain why it's working for @JurrianFahner now, but not for others. It seems like something has been fixed, but this issue is also caused by #384. This problem has also been reported with Spring Boot, but has not been resolved yet: https://github.com/spring-projects/spring-boot/issues/9331.

When autowiring ResourceLoader (or ApplicationContext), and calling resourceLoader.getResource(location), having Spring Boot Devtools on the classpath, the location is incorrectly resolved as a ServletContextResource with path starting with /s3://. Without Devtools it is correctly resolved to a SimpleStorageResource.

The GenericApplicationContext.getResource method calls it's own ResourceLoader instead of super.getResource. Without Devtools there is no resourceLoader set in GenericApplicationContext, so super.getResource gets called and it works correctly.

When I try to add a S3 resource location in WebMvcConfigurer.addResourceHandlers with ResourceHandlerRegistry.addResourceLocations, the same happens:

ResourceHttpRequestHandler.resolveResourceLocations calls applicationContext.getResource(location) for each location, but with Devtools on the classpath, the GenericApplicationContext.getResource method calls it's own ResourceLoader instead of super.getResource.

[JurrianFahner]

@darioseidl We should maybe list projects which aren't working with specific combinations.
But for me it's working with spring-boot 2.1.4 and greenwich.SR1. I don't have devtools on my classpath, so that's no problem for me.

I've changed the title, because the issue applies now to other situations than finley only.

[annagapuz]

@gysel

I had to manually create a SimpleStorageProtocolResolver by auto-wiring in AmazonS3 to my bean and passing it in to the constructor:

@Autowired
private AmazonS3 amazonS3;

// Spring Core IO ResourceLoader - AWS context XML sets this correctly
// If you are looking to set and configure the AWS context in code, I have no example for that
@Autowired
private ResourceLoader resourceLoader;

public void myMethodToFetchFromS3() {
SimpleStorageProtocolResolver simpleStorageProtocolResolver = new SimpleStorageProtocolResolver(amazonS3);

Resource aResource = resourceLoader.getResource("s3://bucket/path/to/resource");

// The workaround
if (!aResource.getClass().getName().endsWith("SimpleStorageResource") && 
         resourceLoader instanceof DefaultResourceLoader) {
  aResource = simpleStorageProtocolResolver.resolve("s3://bucket/path/to/resource", resourceLoader);
  }
}

Hope that helps.

[stoan]

I get the following error if Devtools is active

java.nio.file.InvalidPathException: Illegal char <:> at index 18: src\main\webapp\s3:\swooop\rootFile.log

    at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
    at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
    at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
    at java.io.File.toPath(File.java:2234)
    at org.springframework.core.io.FileSystemResource.<init>(FileSystemResource.java:82)
    at org.springframework.core.io.FileSystemResourceLoader$FileSystemContextResource.<init>(FileSystemResourceLoader.java:65)
    at org.springframework.core.io.FileSystemResourceLoader.getResourceByPath(FileSystemResourceLoader.java:54)
    at org.springframework.core.io.DefaultResourceLoader.getResource(DefaultResourceLoader.java:168)
    at org.springframework.mock.web.MockServletContext.getResourceAsStream(MockServletContext.java:345)
    at org.springframework.web.context.support.ServletContextResource.getInputStream(ServletContextResource.java:156)
    at com.housescent.sweeep.SweeepCoreApplicationTests.resourceLoadingMethod(SweeepCoreApplicationTests.java:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
    at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

@RunWith(SpringRunner.class)
@SpringBootTest
public class SweeepCoreApplicationTests {

    @Autowired
    private ResourceLoader resourceLoader;

    @Test
    public void contextLoads() {
    }

    @Test
    public void resourceLoadingMethod() throws IOException {
        Resource resource = this.resourceLoader.getResource("s3://swooop/rootFile.log");
        Resource secondResource = this.resourceLoader.getResource("s3://myBucket/rootFolder/subFile");

        InputStream inputStream = resource.getInputStream();
        //read file
    }

}

[denis111]

The problem just returned to me with Spring Boot 2.2.1 (maybe 2.2.0 also). But my previous workaround didn't work so I used workaround by @annagapuz but instantiating SimpleStorageProtocolResolver only once in the constructor of my bean.

[annagapuz]

The problem just returned to me with Spring Boot 2.2.1 (maybe 2.2.0 also). But my previous workaround didn't work so I used workaround by @annagapuz but instantiating SimpleStorageProtocolResolver only once in the constructor of my bean.

Good to know - about to upgrade to spring boot 2.2.x

[JurrianFahner]

@denis111 and @annagapuz It is nice that you both are testing spring cloud aws on spring-boot 2.2.x. But were you also aware that spring-cloud is not yet ready for spring boot 2.2.x (when you reported the bugs)? I can't figure it out from your feedback, which spring-cloud version you used.
Spring cloud uses a release train, as you can learn from: https://spring.io/projects/spring-cloud

Release Train | Boot Version
-- | --
Hoxton | 2.2.x
Greenwich | 2.1.x
Finchley | 2.0.x
Edgware | 1.5.x
Dalston | 1.5.x

So in order to get spring-boot 2.2.x working with spring cloud, you'll need Hoxton.RELEASE. Can you please let us know whether the problem persists when you upgrade to Hoxton.RELEASE?

[darioseidl]

This seems to be fixed with Spring Boot 2.2.1 and Spring Cloud AWS 2.2.0 for me. The bug, where the SimpleStorageProtocolResolver would not be resolved when DevTools was on the classpath, has been fixed in Spring Boot and Spring Framework: https://github.com/spring-projects/spring-boot/issues/17214

I have a simple test here: https://github.com/darioseidl/s3demo. With and without Spring Boot Devtools on the classpath, the SimpleStorageResource is resolved correctly now.

[denis111]

@JurrianFahner yes, I was on Greenwich and can confirm that with Hoxton I don't need the workaround.

[denis111]

No, It just happend to me again running with Gradle's bootRun java.io.FileNotFoundException: Could not open ServletContext resource [/s3://...
But worken one time running from IDE.

[annagapuz]

@JurrianFahner I saw the error with Boot 2.1.3-RELEASE and Cloud Greenwich.RELEASE. I have not tested with 2.2.X + Hoxton. Will be doing an upgrade soon though.

[artem-emelin]

Ran into the issue with Boot 2.2.2.RELEASE + Cloud Hoxton.SR1
Workaround by @annagapuz helped

[maciejwalkowiak]

I am not able to reproduce this bug with Spring Boot 2.3 and Spring Cloud AWS 2.2.2.

If you are still facing this issue please comment ideally with a link to github repo that reproduces the bug.

[PavelPolyakov]

@maciejwalkowiak here is a repo with the latest versions, where s3 integration from the guide doesn't work. Unless custom resource loader is defined.

https://github.com/PP-etc/spring-boot-with-s3-example/blob/master/src/main/java/com/pavelpolyakov/s3example/S3ExampleApplication.java#L33

When it's not available, I have this exception:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 's3ExampleApplication': Invocation of init method failed; nested exception is java.lang.ClassCastException: class org.springframework.web.context.support.ServletContextResource cannot be cast to class org.springframework.core.io.WritableResource (org.springframework.web.context.support.ServletContextResource and org.springframework.core.io.WritableResource are in unnamed module of loader 'app')

[maciejwalkowiak]

Thanks @PavelPolyakov! We will look into it.