Django-storages: In django-storages library why safe_join function is not imported in s3boto3.py file from storages.utils.py file

[rahul6612]

Currently i am using Django(version=1.8.4), django-storages(version=1.6.5) and boto3(version=1.5.35). while running the command heroku run python manage.py collectstatic i am getting this error:

from storages.utils import safe_join, setting

ImportError: cannot import name 'safe_join'

here is our safe_join function is defined in utils.py file

`def safe_join(base, *paths):
"""
A version of django.utils._os.safe_join for S3 paths.

Joins one or more path components to the base path component
intelligently. Returns a normalized version of the final path.

The final path must be located inside of the base path component
(otherwise a ValueError is raised).

Paths outside the base path indicate a possible security
sensitive operation.
"""
base_path = force_text(base)
base_path = base_path.rstrip('/')
paths = [force_text(p) for p in paths]

final_path = posixpath.normpath(posixpath.join(base_path + '/', *paths))
# posixpath.normpath() strips the trailing /. Add it back.
if paths[-1].endswith('/'):
    final_path += '/'

# Ensure final_path starts with base_path and that the next character after
# the final path is /.
base_path_len = len(base_path)
if (not final_path.startswith(base_path) or final_path[base_path_len] != '/'):
    raise ValueError('the joined path is located outside of the base path'
                     ' component')

return final_path.lstrip('/')`

and here we imported safe_join function in s3boto3.py file
from storages.utils import setting, safe_join

[jschneier]

That function was moved to that file in version 1.6

Are you sure you don't have a legacy copy of django-storages hanging around somehow?

[rahul6612]

i checked different version of django-storages like 1.4, 1.5, 1.7 on all versions i am getting the same error
and in my python site-packages directory has only one storages directory is there that contains utils.py file and s3boto3.py file
i am trying to fix it from 3 days still didn't get the solution so please help

[jschneier]

Can you post the full traceback?

On Sep 29, 2018, at 6:20 PM, rahul6612 [e-mail] wrote:

i checked different version of django-storages like 1.4, 1.5, 1.7 on all versions i am getting the same error
and in my python site-packages directory only one storages directory is there that contains utils.py file and s3boto3.py file
i am trying to fix it from 3 days still didn't get the solution so please help

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/jschneier/django-storages/issues/608#issuecomment-425679150, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJB2MUoH3dTMW0kFl_ZeClP_SmPtjW9ks5uf_IVgaJpZM4XAr8D.

[rahul6612]

inside storages directory utils.py file
`
import os
import posixpath

from django.conf import settings
from django.core.exceptions import (
ImproperlyConfigured, SuspiciousFileOperation,
)
from django.utils.encoding import force_text

def setting(name, default=None):
return getattr(settings, name, default)

def clean_name(name):
clean_name = posixpath.normpath(name).replace('\', '/')

if name.endswith('/') and not clean_name.endswith('/'):
    # Add a trailing slash as it was stripped.
    clean_name = clean_name + '/'


if clean_name == '.':
    clean_name = ''

return clean_name

def safe_join(base, *paths):
base_path = force_text(base)
base_path = base_path.rstrip('/')
paths = [force_text(p) for p in paths]

final_path = base_path + '/'
for path in paths:
    _final_path = posixpath.normpath(posixpath.join(final_path, path))

    if path.endswith('/') or _final_path + '/' == final_path:
        _final_path += '/'
    final_path = _final_path
if final_path == base_path:
    final_path += '/'


base_path_len = len(base_path)
if (not final_path.startswith(base_path) or final_path[base_path_len] != '/'):
    raise ValueError('the joined path is located outside of the base path'
                     ' component')

return final_path.lstrip('/')

def check_location(storage):
if storage.location.startswith('/'):
correct = storage.location.lstrip('/')
raise ImproperlyConfigured(
"%s.location cannot begin with a leading slash. Found '%s'. Use '%s' instead." % (
storage.__class__.__name__,
storage.location,
correct,
)
)

def lookup_env(names):
for name in names:
value = os.environ.get(name)
if value:
return value

def get_available_overwrite_name(name, max_length):
if max_length is None or len(name) < max_length:
return name

dir_name, file_name = os.path.split(name)
file_root, file_ext = os.path.splitext(file_name)
truncation = len(name) - max_length

file_root = file_root[:-truncation]
if not file_root:
    raise SuspiciousFileOperation(
        'Storage tried to truncate away entire filename "%s". '
        'Please make sure that the corresponding file field '
        'allows sufficient "max_length".' % name
    )
return os.path.join(dir_name, "%s%s" % (file_root, file_ext))

`

inside storages ->backends directory s3boto3.py file

`import mimetypes
import os
import posixpath
import threading
import warnings
from gzip import GzipFile
from tempfile import SpooledTemporaryFile

from django.conf import settings as django_settings
from django.core.exceptions import ImproperlyConfigured, SuspiciousOperation
from django.core.files.base import File
from django.core.files.storage import Storage
from django.utils.deconstruct import deconstructible
from django.utils.encoding import (
filepath_to_uri, force_bytes, force_text, smart_text,
)
from django.utils.six import BytesIO
from django.utils.six.moves.urllib import parse as urlparse
from django.utils.timezone import is_naive, localtime

from storages.utils import (
check_location, get_available_overwrite_name, lookup_env, safe_join,
setting,
)

try:
import boto3.session
from boto3 import __version__ as boto3_version
from botocore.client import Config
from botocore.exceptions import ClientError
except ImportError:
raise ImproperlyConfigured("Could not load Boto3's S3 bindings.\n"
"See https://github.com/boto/boto3")

boto3_version_info = tuple([int(i) for i in boto3_version.split('.')])

@deconstructible
class S3Boto3StorageFile(File):
buffer_size = setting('AWS_S3_FILE_BUFFER_SIZE', 5242880)

def __init__(self, name, mode, storage, buffer_size=None):
    if 'r' in mode and 'w' in mode:
        raise ValueError("Can't combine 'r' and 'w' in mode.")
    self._storage = storage
    self.name = name[len(self._storage.location):].lstrip('/')
    self._mode = mode
    self.obj = storage.bucket.Object(storage._encode_name(name))
    if 'w' not in mode:
        self.obj.load()
    self._is_dirty = False
    self._file = None
    self._multipart = None
    if buffer_size is not None:
        self.buffer_size = buffer_size
    self._write_counter = 0

@property
def size(self):
    return self.obj.content_length

def _get_file(self):
    if self._file is None:
        self._file = SpooledTemporaryFile(
            max_size=self._storage.max_memory_size,
            suffix=".S3Boto3StorageFile",
            dir=setting("FILE_UPLOAD_TEMP_DIR")
        )
        if 'r' in self._mode:
            self._is_dirty = False
            self.obj.download_fileobj(self._file)
            self._file.seek(0)
        if self._storage.gzip and self.obj.content_encoding == 'gzip':
            self._file = GzipFile(mode=self._mode, fileobj=self._file, mtime=0.0)
    return self._file

def _set_file(self, value):
    self._file = value

file = property(_get_file, _set_file)

def read(self, *args, **kwargs):
    if 'r' not in self._mode:
        raise AttributeError("File was not opened in read mode.")
    return super(S3Boto3StorageFile, self).read(*args, **kwargs)

def write(self, content):
    if 'w' not in self._mode:
        raise AttributeError("File was not opened in write mode.")
    self._is_dirty = True
    if self._multipart is None:
        parameters = self._storage.object_parameters.copy()
        if self._storage.default_acl:
            parameters['ACL'] = self._storage.default_acl
        parameters['ContentType'] = (mimetypes.guess_type(self.obj.key)[0] or
                                     self._storage.default_content_type)
        if self._storage.reduced_redundancy:
            parameters['StorageClass'] = 'REDUCED_REDUNDANCY'
        if self._storage.encryption:
            parameters['ServerSideEncryption'] = 'AES256'
        self._multipart = self.obj.initiate_multipart_upload(**parameters)
    if self.buffer_size <= self._buffer_file_size:
        self._flush_write_buffer()
    return super(S3Boto3StorageFile, self).write(force_bytes(content))

@property
def _buffer_file_size(self):
    pos = self.file.tell()
    self.file.seek(0, os.SEEK_END)
    length = self.file.tell()
    self.file.seek(pos)
    return length

def _flush_write_buffer(self):
    if self._buffer_file_size:
        self._write_counter += 1
        self.file.seek(0)
        part = self._multipart.Part(self._write_counter)
        part.upload(Body=self.file.read())
        self.file.seek(0)
        self.file.truncate()

def close(self):
    if self._is_dirty:
        self._flush_write_buffer()
        parts = [{'ETag': part.e_tag, 'PartNumber': part.part_number}
                 for part in self._multipart.parts.all()]
        self._multipart.complete(
            MultipartUpload={'Parts': parts})
    else:
        if self._multipart is not None:
            self._multipart.abort()
    if self._file is not None:
        self._file.close()
        self._file = None

@deconstructible
class S3Boto3Storage(Storage):
default_content_type = 'application/octet-stream'

config = None


access_key_names = ['AWS_S3_ACCESS_KEY_ID', 'AWS_ACCESS_KEY_ID']
secret_key_names = ['AWS_S3_SECRET_ACCESS_KEY', 'AWS_SECRET_ACCESS_KEY']
security_token_names = ['AWS_SESSION_TOKEN', 'AWS_SECURITY_TOKEN']
security_token = None

access_key = setting('AWS_S3_ACCESS_KEY_ID', setting('AWS_ACCESS_KEY_ID'))
secret_key = setting('AWS_S3_SECRET_ACCESS_KEY', setting('AWS_SECRET_ACCESS_KEY'))
file_overwrite = setting('AWS_S3_FILE_OVERWRITE', True)
object_parameters = setting('AWS_S3_OBJECT_PARAMETERS', {})
bucket_name = setting('AWS_STORAGE_BUCKET_NAME')
auto_create_bucket = setting('AWS_AUTO_CREATE_BUCKET', False)
default_acl = setting('AWS_DEFAULT_ACL', 'public-read')
bucket_acl = setting('AWS_BUCKET_ACL', default_acl)
querystring_auth = setting('AWS_QUERYSTRING_AUTH', True)
querystring_expire = setting('AWS_QUERYSTRING_EXPIRE', 3600)
signature_version = setting('AWS_S3_SIGNATURE_VERSION')
reduced_redundancy = setting('AWS_REDUCED_REDUNDANCY', False)
location = setting('AWS_LOCATION', '')
encryption = setting('AWS_S3_ENCRYPTION', False)
custom_domain = setting('AWS_S3_CUSTOM_DOMAIN')
addressing_style = setting('AWS_S3_ADDRESSING_STYLE')
secure_urls = setting('AWS_S3_SECURE_URLS', True)
file_name_charset = setting('AWS_S3_FILE_NAME_CHARSET', 'utf-8')
gzip = setting('AWS_IS_GZIPPED', False)
preload_metadata = setting('AWS_PRELOAD_METADATA', False)
gzip_content_types = setting('GZIP_CONTENT_TYPES', (
    'text/css',
    'text/javascript',
    'application/javascript',
    'application/x-javascript',
    'image/svg+xml',
))
url_protocol = setting('AWS_S3_URL_PROTOCOL', 'http:')
endpoint_url = setting('AWS_S3_ENDPOINT_URL')
proxies = setting('AWS_S3_PROXIES')
region_name = setting('AWS_S3_REGION_NAME')
use_ssl = setting('AWS_S3_USE_SSL', True)
verify = setting('AWS_S3_VERIFY', None)
max_memory_size = setting('AWS_S3_MAX_MEMORY_SIZE', 0)

def __init__(self, acl=None, bucket=None, **settings):
    for name, value in settings.items():
        if hasattr(self, name):
            setattr(self, name, value)


    if acl is not None:
        warnings.warn(
            "The acl argument of S3Boto3Storage is deprecated. Use "
            "argument default_acl or setting AWS_DEFAULT_ACL instead. The "
            "acl argument will be removed in version 2.0.",
            DeprecationWarning,
        )
        self.default_acl = acl
    if bucket is not None:
        warnings.warn(
            "The bucket argument of S3Boto3Storage is deprecated. Use "
            "argument bucket_name or setting AWS_STORAGE_BUCKET_NAME "
            "instead. The bucket argument will be removed in version 2.0.",
            DeprecationWarning,
        )
        self.bucket_name = bucket

    check_location(self)


    if self.secure_urls:
        self.url_protocol = 'https:'

    self._entries = {}
    self._bucket = None
    self._connections = threading.local()

    self.access_key, self.secret_key = self._get_access_keys()
    self.security_token = self._get_security_token()

    if not self.config:
        kwargs = dict(
            s3={'addressing_style': self.addressing_style},
            signature_version=self.signature_version,
        )

        if boto3_version_info >= (1, 4, 4):
            kwargs['proxies'] = self.proxies
        else:
            warnings.warn(
                "In version 2.0 of django-storages the minimum required version of "
                "boto3 will be 1.4.4. You have %s " % boto3_version_info
            )
        self.config = Config(**kwargs)


    if not hasattr(django_settings, 'AWS_DEFAULT_ACL'):
        warnings.warn(
            "The default behavior of S3Boto3Storage is insecure and will change "
            "in django-storages 2.0. By default files and new buckets are saved "
            "with an ACL of 'public-read' (globally publicly readable). Version 2.0 will "
            "default to using the bucket's ACL. To opt into the new behavior set "
            "AWS_DEFAULT_ACL = None, otherwise to silence this warning explicitly "
            "set AWS_DEFAULT_ACL."
        )

def __getstate__(self):
    state = self.__dict__.copy()
    state.pop('_connections', None)
    state.pop('_bucket', None)
    return state

def __setstate__(self, state):
    state['_connections'] = threading.local()
    state['_bucket'] = None
    self.__dict__ = state

@property
def connection(self):
    connection = getattr(self._connections, 'connection', None)
    if connection is None:
        session = boto3.session.Session()
        self._connections.connection = session.resource(
            's3',
            aws_access_key_id=self.access_key,
            aws_secret_access_key=self.secret_key,
            aws_session_token=self.security_token,
            region_name=self.region_name,
            use_ssl=self.use_ssl,
            endpoint_url=self.endpoint_url,
            config=self.config,
            verify=self.verify,
        )
    return self._connections.connection

@property
def bucket(self):
    if self._bucket is None:
        self._bucket = self._get_or_create_bucket(self.bucket_name)
    return self._bucket

@property
def entries(self):
    if self.preload_metadata and not self._entries:
        self._entries = {
            self._decode_name(entry.key): entry
            for entry in self.bucket.objects.filter(Prefix=self.location)
        }
    return self._entries

def _get_access_keys(self):
    access_key = self.access_key or lookup_env(S3Boto3Storage.access_key_names)
    secret_key = self.secret_key or lookup_env(S3Boto3Storage.secret_key_names)
    return access_key, secret_key

def _get_security_token(self):
    security_token = self.security_token or lookup_env(S3Boto3Storage.security_token_names)
    return security_token

def _get_or_create_bucket(self, name):
    bucket = self.connection.Bucket(name)
    if self.auto_create_bucket:
        try:
            bucket.meta.client.head_bucket(Bucket=name)
        except ClientError as err:
            if err.response['ResponseMetadata']['HTTPStatusCode'] == 301:
                raise ImproperlyConfigured("Bucket %s exists, but in a different "
                                           "region than we are connecting to. Set "
                                           "the region to connect to by setting "
                                           "AWS_S3_REGION_NAME to the correct region." % name)

            elif err.response['ResponseMetadata']['HTTPStatusCode'] == 404:
                if not hasattr(django_settings, 'AWS_BUCKET_ACL'):
                    warnings.warn(
                        "The default behavior of S3Boto3Storage is insecure and will change "
                        "in django-storages 2.0. By default new buckets are saved with an ACL of "
                        "'public-read' (globally publicly readable). Version 2.0 will default to "
                        "Amazon's default of the bucket owner. To opt into this behavior this warning "
                        "set AWS_BUCKET_ACL = None, otherwise to silence this warning explicitly set "
                        "AWS_BUCKET_ACL."
                    )
                if self.bucket_acl:
                    bucket_params = {'ACL': self.bucket_acl}
                else:
                    bucket_params = {}
                region_name = self.connection.meta.client.meta.region_name
                if region_name != 'us-east-1':
                    bucket_params['CreateBucketConfiguration'] = {
                        'LocationConstraint': region_name}
                bucket.create(**bucket_params)
            else:
                raise
    return bucket

def _clean_name(self, name):

    clean_name = posixpath.normpath(name).replace('\\', '/')
    if name.endswith('/') and not clean_name.endswith('/'):

        clean_name += '/'
    return clean_name

def _normalize_name(self, name):
    try:
        return safe_join(self.location, name)
    except ValueError:
        raise SuspiciousOperation("Attempted access to '%s' denied." %
                                  name)

def _encode_name(self, name):
    return smart_text(name, encoding=self.file_name_charset)

def _decode_name(self, name):
    return force_text(name, encoding=self.file_name_charset)

def _compress_content(self, content):
    """Gzip a given string content."""
    content.seek(0)
    zbuf = BytesIO()

    zfile = GzipFile(mode='wb', fileobj=zbuf, mtime=0.0)
    try:
        zfile.write(force_bytes(content.read()))
    finally:
        zfile.close()
    zbuf.seek(0)

    return zbuf

def _open(self, name, mode='rb'):
    name = self._normalize_name(self._clean_name(name))
    try:
        f = S3Boto3StorageFile(name, mode, self)
    except ClientError as err:
        if err.response['ResponseMetadata']['HTTPStatusCode'] == 404:
            raise IOError('File does not exist: %s' % name)
        raise  
    return f

def _save(self, name, content):
    cleaned_name = self._clean_name(name)
    name = self._normalize_name(cleaned_name)
    parameters = self.object_parameters.copy()
    _type, encoding = mimetypes.guess_type(name)
    content_type = getattr(content, 'content_type', None)
    content_type = content_type or _type or self.default_content_type


    parameters.update({'ContentType': content_type})

    if self.gzip and content_type in self.gzip_content_types:
        content = self._compress_content(content)
        parameters.update({'ContentEncoding': 'gzip'})
    elif encoding:

        parameters.update({'ContentEncoding': encoding})

    encoded_name = self._encode_name(name)
    obj = self.bucket.Object(encoded_name)
    if self.preload_metadata:
        self._entries[encoded_name] = obj


    if isinstance(content, File):
        content = content.file

    self._save_content(obj, content, parameters=parameters)
    return cleaned_name

def _save_content(self, obj, content, parameters):
    put_parameters = parameters.copy() if parameters else {}
    if self.encryption:
        put_parameters['ServerSideEncryption'] = 'AES256'
    if self.reduced_redundancy:
        put_parameters['StorageClass'] = 'REDUCED_REDUNDANCY'
    if self.default_acl:
        put_parameters['ACL'] = self.default_acl
    content.seek(0, os.SEEK_SET)
    obj.upload_fileobj(content, ExtraArgs=put_parameters)

def delete(self, name):
    name = self._normalize_name(self._clean_name(name))
    self.bucket.Object(self._encode_name(name)).delete()

def exists(self, name):
    name = self._normalize_name(self._clean_name(name))
    if self.entries:
        return name in self.entries
    try:
        self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
        return True
    except ClientError:
        return False

def listdir(self, name):
    path = self._normalize_name(self._clean_name(name))
    if path and not path.endswith('/'):
        path += '/'

    directories = []
    files = []
    paginator = self.connection.meta.client.get_paginator('list_objects_v2')
    pages = paginator.paginate(Bucket=self.bucket_name, Delimiter='/', Prefix=path)
    for page in pages:
        for entry in page.get('CommonPrefixes', ()):
            directories.append(posixpath.relpath(entry['Prefix'], path))
        for entry in page.get('Contents', ()):
            files.append(posixpath.relpath(entry['Key'], path))
    return directories, files

def size(self, name):
    name = self._normalize_name(self._clean_name(name))
    if self.entries:
        entry = self.entries.get(name)
        if entry:
            return entry.size if hasattr(entry, 'size') else entry.content_length
        return 0
    return self.bucket.Object(self._encode_name(name)).content_length

def get_modified_time(self, name):
    name = self._normalize_name(self._clean_name(name))
    entry = self.entries.get(name)
    if entry is None:
        entry = self.bucket.Object(self._encode_name(name))
    if setting('USE_TZ'):
        return entry.last_modified
    else:
        return localtime(entry.last_modified).replace(tzinfo=None)

def modified_time(self, name):
    mtime = self.get_modified_time(name)
    return mtime if is_naive(mtime) else localtime(mtime).replace(tzinfo=None)

def _strip_signing_parameters(self, url):
    split_url = urlparse.urlsplit(url)
    qs = urlparse.parse_qsl(split_url.query, keep_blank_values=True)
    blacklist = {
        'x-amz-algorithm', 'x-amz-credential', 'x-amz-date',
        'x-amz-expires', 'x-amz-signedheaders', 'x-amz-signature',
        'x-amz-security-token', 'awsaccesskeyid', 'expires', 'signature',
    }
    filtered_qs = ((key, val) for key, val in qs if key.lower() not in blacklist)
    joined_qs = ('='.join(keyval) for keyval in filtered_qs)
    split_url = split_url._replace(query="&".join(joined_qs))
    return split_url.geturl()

def url(self, name, parameters=None, expire=None):
    name = self._normalize_name(self._clean_name(name))
    if self.custom_domain:
        return "%s//%s/%s" % (self.url_protocol,
                              self.custom_domain, filepath_to_uri(name))
    if expire is None:
        expire = self.querystring_expire

    params = parameters.copy() if parameters else {}
    params['Bucket'] = self.bucket.name
    params['Key'] = self._encode_name(name)
    url = self.bucket.meta.client.generate_presigned_url('get_object', Params=params,
                                                         ExpiresIn=expire)
    if self.querystring_auth:
        return url
    return self._strip_signing_parameters(url)

def get_available_name(self, name, max_length=None):
    """Overwrite existing file with the same name."""
    name = self._clean_name(name)
    if self.file_overwrite:
        return get_available_overwrite_name(name, max_length)
    return super(S3Boto3Storage, self).get_available_name(name, max_length)

**here i am using these library in my django project i created a file custom_storages.py at the same label of manage.py file**
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage

class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION

class MediaStorage(S3Boto3Storage):
location = settings.MEDIAFILES_LOCATION`

and here is my main settings.py file is
`STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'

MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'

AWS_STORAGE_BUCKET_NAME = 'darkmachine'
AWS_S3_REGION_NAME = 'ap-south-1' # e.g. us-east-2
AWS_ACCESS_KEY_ID = 'xxxxxxxxxxxxxxxxx'
AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME`

[sww314]

@rahul6612 did you figure this out?

[rahul6612]

ya i got that