Imageprocessor: CloudImageService and AzureBlobCache with private storage

[tobbbe]

Description

Is it possible to use CloudImageService and AzureBlobCache with a Private azure blob container (using UmbracoFileSystemProviders.Azure)?
Everything works great if I use "Blob" as Public access (in azure portal) level, but if I set it to "Private" and usePrivateContainer to "true" in FileSystemProvider.config this is the response (404):

<Error>
    <Code>
        ResourceNotFound
    </Code>
    <Message>
        The specified resource does not exist.
    RequestId:2d014e79-001e-00a4-450f-41ce81000000
    Time:2018-08-31T09:44:01.4799466Z
    </Message>
</Error>

Im trying to serve protected umbraco media files for a closed intranet.
Any help is appreciated!

System Configuration

• ImageProcessor version: 2.6.2.25
• ImageProcessor.Web version: 4.9.3.25
• Other ImageProcessor packages and versions:
  ImageProcessor.Web.Config: 2.4.1.19
  ImageProcessor.Web.Plugins.AzureBlobCache: 1.4.2.19
  UmbracoFileSystemProviders.Azure: 1.0.2
• Environment (Operating system, version and so on): Umbraco cloud / Windows 10 (local dev)
• .NET Framework version: 4.5.2

security.config

<?xml version="1.0" encoding="utf-8"?>
<security>

<services>
    <service name="CloudImageService" prefix="media/" type="ImageProcessor.Web.Services.CloudImageService, ImageProcessor.Web">
      <settings>
        <setting key="MaxBytes" value="8194304"/>
        <setting key="Timeout" value="30000"/>
        <setting key="Host" value="https://name.blob.core.windows.net/media/"/>
      </settings>
    </service>
  </services></security>

cache.config

<?xml version="1.0" encoding="utf-8"?>
<caching currentCache="AzureBlobCache">
  <caches>

    <cache name="AzureBlobCache" type="ImageProcessor.Web.Plugins.AzureBlobCache.AzureBlobCache, ImageProcessor.Web.Plugins.AzureBlobCache" maxDays="365">
      <settings>
        <setting key="CachedStorageAccount" value="DefaultEndpointsProtocol=https;AccountName=name;AccountKey=key" />
        <setting key="CachedBlobContainer" value="cache" />
        <setting key="CachedCDNRoot" value="https://name.blob.core.windows.net" />
        <setting key="CachedCDNTimeout" value="2000" />
        <setting key="UseCachedContainerInUrl" value="true" />
        <setting key="SourceStorageAccount" value="DefaultEndpointsProtocol=https;AccountName=name;AccountKey=" />
        <setting key="SourceBlobContainer" value="media" />
        <setting key="StreamCachedImage" value="true" />
      </settings>
    </cache>

  </caches>
</caching>

FileSystemProvider.config

<?xml version="1.0"?>
<FileSystemProviders>

  <Provider alias="media" type="Our.Umbraco.FileSystemProviders.Azure.AzureBlobFileSystem, Our.Umbraco.FileSystemProviders.Azure">
    <Parameters>
        <add key="containerName" value="media"/>
        <add key="rootUrl" value="https://name.blob.core.windows.net/"/>
        <add key="connectionString" value="DefaultEndpointsProtocol=https;AccountName=name;AccountKey=key;EndpointSuffix=core.windows.net"/>

        <add key="maxDays" value="365"/>

        <add key="useDefaultRoute" value="true"/>

        <add key="usePrivateContainer" value="true"/>
    </Parameters>
  </Provider>

</FileSystemProviders>

[idseefeld]

Hi tobbbe,
the error is in file "security.config" for "Host" setting you have to enter the base URL for the Umbraco site and not the Azure Storage. E.g. "http://mySite" without trailing slash and without /media at the end. The FileSystemProvider will take care of the blob access.
By the way the "UsePrivateContainer" config setting for the AzureBlobFileSystem provider will toggle the blob access automatically. You don't have to do it in the Azure Portal.
You can use both settings for "StreamCachedImage" in \config\imageprocessor\cache.config as the "cache" container should always be public. If you actually want to use Azures protection for private blobs you have to think about using SAS token. But these are currently not supported and such a solution might be a little tricky.
Yours Dirk

[idseefeld]

Sorry,
my last comment does not seem to work very well. Probably it would be necessary to create a new CloudImageService which uses the Azure API directly...
But as already mentioned a proper use of private blobs will take much more effort.
Yours Dirk

[JimBobSquarePants]

Hi @tobbbe

You'd need to create your own IImageService implementation since CloudImageService is essentially a HttpRequest.

The FileSystemProvider is an Umbraco specific API that ImageProcessor.Web has no knowledge of so changing settings there will have no effect.

@idseefeld Your approach would not work since the ImageProcessingModule needs the request to come from an IImageService.

I had 5 minutes just there so a put together a completely untested one for you.

<!--You can delete the prefix attribute if you are not serving any local files and want to match all image requests with commands.-->
<service prefix="media/" name="AzureImageService" type="[Namespace].AzureImageService, [AssemblyName]">
  <settings>
    <setting key="StorageAccount" value="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]"/>
    <setting key="Container" value="media"/>
    <setting key="AccessType" value="Off"/>
  </settings>
</service>

```c#
///

/// An image service for retrieving images from Azure.
///

public class AzureImageService : IImageService
{
private CloudBlobContainer blobContainer;
private Dictionary settings = new Dictionary();

/// <summary>
/// Gets or sets the prefix for the given implementation.
/// <remarks>
/// This value is used as a prefix for any image requests that should use this service.
/// </remarks>
/// </summary>
public string Prefix { get; set; } = string.Empty;

/// <summary>
/// Gets a value indicating whether the image service requests files from
/// the locally based file system.
/// </summary>
public bool IsFileLocalService => false;

/// <summary>
/// Gets or sets any additional settings required by the service.
/// </summary>
public Dictionary<string, string> Settings
{
    get => this.settings;
    set
    {
        this.settings = value;
        this.InitService();
    }
}

/// <summary>
/// Gets or sets the white list of <see cref="Uri" />. 
/// </summary>
public Uri[] WhiteList { get; set; }

/// <summary>
/// Gets the image using the given identifier.
/// </summary>
/// <param name="id">The value identifying the image to fetch.</param>
/// <returns>
/// The <see cref="byte" /> array containing the image data.
/// </returns>
public async Task<byte[]> GetImage(object id)
{
    CloudBlockBlob blockBlob = this.blobContainer.GetBlockBlobReference(id.ToString());

    if (blockBlob.Exists())
    {
        using (MemoryStream memoryStream = MemoryStreamPool.Shared.GetStream())
        {
            await blockBlob.DownloadToStreamAsync(memoryStream).ConfigureAwait(false);
            return memoryStream.ToArray();
        }
    }

    return null;
}

/// <summary>
/// Gets a value indicating whether the current request passes sanitizing rules.
/// </summary>
/// <param name="path">The image path.</param>
/// <returns>
/// <c>True</c> if the request is valid; otherwise, <c>False</c>.
/// </returns>
public bool IsValidRequest(string path) => ImageHelpers.IsValidImageExtension(path);

/// <summary>
/// Initialise the service.
/// </summary>
private void InitService()
{
    // Retrieve storage accounts from connection string.
    var cloudCachedStorageAccount = CloudStorageAccount.Parse(this.Settings["StorageAccount"]);

    // Create the blob client.
    CloudBlobClient blobClient = cloudCachedStorageAccount.CreateCloudBlobClient();

    string container = this.Settings.ContainsKey("Container")
        ? this.Settings["Container"]
        : string.Empty;

    BlobContainerPublicAccessType accessType = this.Settings.ContainsKey("AccessType")
        ? (BlobContainerPublicAccessType)Enum.Parse(typeof(BlobContainerPublicAccessType), this.Settings["AccessType"])
        : BlobContainerPublicAccessType.Blob;

    this.blobContainer = CreateContainer(blobClient, container, accessType);
}

/// <summary>
/// Returns the cache container, creating a new one if none exists.
/// </summary>
/// <param name="cloudBlobClient"><see cref="CloudBlobClient"/> where the container is stored.</param>
/// <param name="containerName">The name of the container.</param>
/// <param name="accessType"><see cref="BlobContainerPublicAccessType"/> indicating the access permissions.</param>
/// <returns>The <see cref="CloudBlobContainer"/></returns>
private static CloudBlobContainer CreateContainer(CloudBlobClient cloudBlobClient, string containerName, BlobContainerPublicAccessType accessType)
{
    CloudBlobContainer container = cloudBlobClient.GetContainerReference(containerName);

    if (!container.Exists())
    {
        container.Create();
        container.SetPermissions(new BlobContainerPermissions { PublicAccess = accessType });
    }

    return container;
}

}
```

You'll need the following Nuget packages.

• WindowsAzure.Storage version 4.3.0
• Microsoft.IO.RecyclableMemoryStream 1.2.2

[tobbbe]

WOW! Works like a charm if I use DiskCache and not AzureBlobCache (getting 404). Thank you so much!!

Now I just have to figure out how to create a IImageCache for private containers. Is it possible to config ImageProcessor.Web.Plugins.AzureBlobCache to my needs or should I try to to modify https://github.com/JimBobSquarePants/ImageProcessor/blob/58e18acd42de65b02421a85dee5361c4d74ac2ff/src/ImageProcessor.Web/Caching/DiskCache.cs with CloudBlobContainer? My cache.config still is like the one in my first post.

Again thank you very much!

EDIT:
If I use ImageProcessor.Web.Plugins.AzureBlobCache the cache is still created in the azure container (private). I guess it returns 404 because AzureBlobCache tries to get the cached image with a http request?

[JimBobSquarePants]

@tobbbe

Happy to help, I had the time and it was not too painful to do.

If I use ImageProcessor.Web.Plugins.AzureBlobCache the cache is still created in the azure container (private). I guess it returns 404 because AzureBlobCache tries to get the cached image with a http request?

Yeah that's exactly right. I'll do some experimenting when I can to see if I can serve private blobs. I might well add the service to the AzureBlobCache Nuget package once I get it all working.

[idseefeld]

Hi James, I think the following doc might help: https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1Would be nice to make the validity duration for the token configurable;-) 

[idseefeld]

Please check out: https://github.com/JimBobSquarePants/ImageProcessor/pull/704

[djanjicek]

James, I've tried the AzureImageService and for some reason the GetImage method is never called. The service itself gets initialized though. I've left the Prefix blank and my image url's are rendered like http://my.cdn-endpoint.com/my-container/my/image/path/xyz.jpg

What might I be missing?

[djanjicek]

Alright, I got it work when using relative image URL's plus a prefix. However it only works with disk storage. Once I switch to azure blob storage I get 404's.

Another question, can we configure the service somehow to strip the query string from the URL once everything is processed?

[JimBobSquarePants]

Alright, I got it work when using relative image URL's plus a prefix. However it only works with disk storage. Once I switch to azure blob storage I get 404's.

@djanjicek That is noted above already and expected since the cache performs a redirect which requires public access.

https://github.com/JimBobSquarePants/ImageProcessor/issues/699#issuecomment-418144529

The current (unpublished as I test) Azure cache codebase actually works well with private blobs as it copies the blob stream to the response body.

Another question, can we configure the service somehow to strip the query string from the URL once everything is processed?

I'm not sure what you mean by this, it doesn't make sense. The request contains the querystring. i.e you make the request. Once a service has responded that request is in the past.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JimBobSquarePants]

Code is in the current dev branch ready for the impending release so will close for now.

[jynzix]

Is the current release capable of private blob storage for both media and cache storage?
I know there's a setting for AccessType which is set to 'Blob' on updating package but switching this to 'Off' doesn't seem to do anything and still returning 404s.
I'm actively trying to avoid using DiskCache to store it locally. I basically want my local dev environment and Azure test slot (which is only available internally) to share the same media/cache store with private storage.

security.config

<?xml version="1.0" encoding="utf-8"?>
<security>
  <services>  
    <service prefix="media/" name="CloudImageService" type="ImageProcessor.Web.Services.CloudImageService, ImageProcessor.Web">
      <settings>
        <setting key="Container" value="media"/>
        <setting key="MaxBytes" value="8194304"/>
        <setting key="Timeout" value="30000"/>
        <setting key="Host" value="[TestMediaHost]"/>
      </settings>
    </service>   
    <service prefix="remote.axd" name="RemoteImageService" type="ImageProcessor.Web.Services.RemoteImageService, ImageProcessor.Web">
      <settings>
        <setting key="MaxBytes" value="4194304" />
        <setting key="Timeout" value="3000" />
        <setting key="Protocol" value="http" />
      </settings>
      <whitelist>
        <add url="..." />
      </whitelist>
    </service>
    <service name="AzureImageService" type="ImageProcessor.Web.Plugins.AzureBlobCache.AzureImageService, ImageProcessor.Web.Plugins.AzureBlobCache">
      <settings>
        <setting key="StorageAccount" value="DefaultEndpointsProtocol=https;AccountName=[CacheAccountName];AccountKey=[CacheAccountKey]" />
        <setting key="Container" value="cache" />
        <setting key="AccessType" value="Off" />
      </settings>
    </service>
    </services>
</security>

cache.config

<?xml version="1.0" encoding="utf-8"?>
<caching currentCache="AzureBlobCache">
  <caches>
    <cache name="AzureBlobCache" type="ImageProcessor.Web.Plugins.AzureBlobCache.AzureBlobCache, ImageProcessor.Web.Plugins.AzureBlobCache" maxDays="365">
        <settings>
          <setting key="CachedStorageAccount" value="[PrivateCachedStorageAccount]" />
          <setting key="CachedBlobContainer" value="cache" />
          <setting key="UseCachedContainerInUrl" value="true" />
          <setting key="SourceStorageAccount" value="[PrivateMediaStorageAccount]" />
          <setting key="SourceBlobContainer" value="media" />
          <setting key="StreamCachedImage" value="false" />
        </settings>
      </cache>
    </caches>
</caching>

[JimBobSquarePants]

You should be using the AzureImageService. Do you need both configured and enabled?

Your Container property in your service is pointed at cache and you have no prefix attribute configured. Is that correct?

Your cache configuration is missing an AccessType property also. Both should be Off not off.

[jynzix]

There's no prefix on the AzureImageServiceabove and the container is cache.
I added in the AccessType for both AzureImageService in the security.config and cache.config and no change.

When I updated AzureBlobCache to 1.7 from 1.5, I'm presented with:

  <service name="AzureImageService" type="ImageProcessor.Web.Plugins.AzureBlobCache.AzureImageService, ImageProcessor.Web.Plugins.AzureBlobCache">
      <settings>
        <setting key="StorageAccount" value="DefaultEndpointsProtocol=https;AccountName=[CacheAccountName];AccountKey=[CacheAccountKey]" />
        <setting key="Container" value="[ContainerName]" />
        <setting key="AccessType" value="Blob" />
      </settings>
    </service>

So I would presume that this service is mainly focused on the cache due to the connection string placeholders (hence why my container name is cache) but I don't intuitively expect that service to replace my CloudImageService as this has reference to my media container and host. Am I right in assuming that CloudImageService is no longer relevant when using AzureImageService?

It might help to mention that I am using UmbracoFileSystemProviders.Azure like tobbbe.

[JimBobSquarePants]

I don't understand. The cache container is where you are saving the converted images not where you are pulling them from.

The purpose of the AzureImageService (and the entire purpose of this thread) is to replace the CloudImageService to allow retrieving private images.

• Your Container property on AzureImageService should match the source container your images are stored in.
• Your AccessType property should be Off for private access to match the BlobContainerPublicAccessType enum.
• Your prefix attribute should match the original CloudImageService value.

The CacheAccountName placeholder exists in that form as the assumption was made you'd use the same account for both source and cache.

Apologies to anyone getting spammed on this closed thread btw.