Aws-sdk-ruby: "Ruby bundle" completion time on aws-sdk exceeds allotted Chef timeout

Created on 16 Mar 2019 · 3Comments · Source: aws/aws-sdk-ruby

Issue description

TLDR: the chef-client command on the aws-sdk dependencies times out before its ruby bundle step has completed.

When invoking a chef run via chef-client, the ruby bundle command is invoked to install dependencies. Problem is, the ruby bundle takes 10:33 minutes to finish. But the chef-client has a 10-minute time limit that is not easily configurable. So the chef update fails.

What is the best way to prevent this from happening?

Gem name

Gemfile:

sudo cat /tmp/chef-gem-bundle20190315-14906-1umgbjn/Gemfile
source 'https://rubygems.org'
gem(*["aws-sdk", "< 4.0", ">= 2"])
gem(*["retries", "~> 0.0"])
gem(*["aws-sdk-cloudformation", "~> 1.13.0"])
gem(*["aws-sdk-cloudwatch", "~> 1.13.0"])
gem(*["aws-sdk-core", "~> 3.44.0"])
gem(*["aws-sdk-dynamodb", "~> 1.18.0"])
gem(*["aws-sdk-ec2", "~> 1.63.0"])
gem(*["aws-sdk-elasticloadbalancing", "~> 1.8.0"])
gem(*["aws-eventstream", "~> 1.0.1"])
gem(*["aws-sdk-iam", "~> 1.13.0"])
gem(*["aws-sdk-kinesis", "~> 1.9.0"])
gem(*["aws-sdk-kms", "~> 1.13.0"])
gem(*["aws-sdk-route53", "~> 1.16.0"])
gem(*["aws-partitions", "~> 1.123.0"])
gem(*["aws-sdk-s3", "~> 1.30.0"])
gem(*["aws-sigv4", "~> 1.0.3"])
gem(*["aws-sdk-ssm", "~> 1.34.0"])

Version of Ruby, OS environment

Ruby version: ruby 2.0.0p648 [x86_64-linux]
OS: Amazon Linux AMI release 2017.09 4.9.76-3.78.amzn1.x86_64
Bundler version: 1.12.5 (We also tried using bundler version 1.17.3, and it still failed).
Chef version: Unfortunately we are currently constrained at version 12.19.36

Code snippets / steps to reproduce

Command:
DEBUG_RESOLVER=1 /opt/chef/embedded/bin/ruby /opt/chef/embedded/bin/bundle install -VV 2>&1 | tee bundle-output.txt
Truncated output included below:

Output:

HTTP GET https://index.rubygems.org/versions
HTTP 304 Not Modified
Fetching gem metadata from https://rubygems.org/
Looking up gems ["aws-sdk", "retries", "aws-sdk-cloudformation", "aws-sdk-cloudwatch", "aws-sdk-core", "aws-sdk-dynamodb", "aws-sdk-ec2", "aws-sdk-elasticloadbalancing", "aws-eventstream", "aws-sdk-iam", "aws-sdk-kinesis", "aws-sdk-kms", "aws-sdk-route53", "aws-partitions", "aws-sdk-s3", "aws-sigv4", "aws-sdk-ssm"]
Looking up gems ["builder", "multi_json", "multi_xml", "jamespath", "jmespath", "http-2", "httparty", "json", "nokogiri", "uuidtools", "aws-sdk-v1", "aws-sdk-resources", "aws-sdk-acm", "aws-sdk-apigateway", "aws-sdk-applicationautoscaling", "aws-sdk-applicationdiscoveryservice", "aws-sdk-autoscaling", "aws-sdk-budgets", "aws-sdk-cloudfront", "aws-sdk-cloudhsm", "aws-sdk-cloudsearch", "aws-sdk-cloudsearchdomain", "aws-sdk-cloudtrail", "aws-sdk-cloudwatchevents", "aws-sdk-cloudwatchlogs", "aws-sdk-codecommit", "aws-sdk-codedeploy", "aws-sdk-codepipeline", "aws-sdk-cognitoidentity", "aws-sdk-cognitoidentityprovider", "aws-sdk-cognitosync", "aws-sdk-configservice", "aws-sdk-databasemigrationservice", "aws-sdk-datapipeline", "aws-sdk-devicefarm", "aws-sdk-directconnect", "aws-sdk-directoryservice", "aws-sdk-dynamodbstreams", "aws-sdk-ecr", "aws-sdk-ecs", "aws-sdk-efs", "aws-sdk-elasticache", "aws-sdk-elasticbeanstalk", "aws-sdk-elasticloadbalancingv2", "aws-sdk-elasticsearchservice", "aws-sdk-elastictranscoder", "aws-sdk-emr", "aws-sdk-firehose", "aws-sdk-gamelift", "aws-sdk-glacier", "aws-sdk-importexport", "aws-sdk-inspector", "aws-sdk-iot", "aws-sdk-iotdataplane", "aws-sdk-kinesisanalytics", "aws-sdk-lambda", "aws-sdk-lambdapreview", "aws-sdk-machinelearning", "aws-sdk-marketplacecommerceanalytics", "aws-sdk-marketplacemetering", "aws-sdk-opsworks", "aws-sdk-rds", "aws-sdk-redshift", "aws-sdk-route53domains", "aws-sdk-servicecatalog", "aws-sdk-ses", "aws-sdk-simpledb", "aws-sdk-sms", "aws-sdk-snowball", "aws-sdk-sns", "aws-sdk-sqs", "aws-sdk-storagegateway", "aws-sdk-support", "aws-sdk-swf", "aws-sdk-waf", "aws-sdk-workspaces", "aws-sdk-appstream", "aws-sdk-codebuild", "aws-sdk-health", "aws-sdk-lightsail", "aws-sdk-opsworkscm", "aws-sdk-pinpoint", "aws-sdk-polly", "aws-sdk-rekognition", "aws-sdk-sfn", "aws-sdk-shield", "aws-sdk-wafregional", "aws-sdk-xray", "aws-sdk-batch", "aws-sdk-clouddirectory", "aws-sdk-costandusagereportservice", "aws-sdk-lexruntimeservice", "aws-sdk-mturk", "aws-sdk-organizations", "aws-sdk-workdocs", "aws-sdk-codestar", "aws-sdk-lexmodelbuildingservice", "aws-sdk-resourcegroupstaggingapi", "aws-sdk-lex", "aws-sdk-marketplaceentitlementservice", "aws-sdk-states", "aws-sdk-athena"]
Looking up gems ["echoe", "jnunemaker-crack", "crack", "activesupport", "hoe", "mime-types", "json_pure", "aws-sigv2", "launchy", "rake", "rspec", "racc", "rake-compiler", "rexical", "tenderlove-frex", "weakling", "mini_portile", "mini_portile2", "pkg-config", "aws-sdk-dax", "aws-sdk-greengrass", "aws-sdk-cloudhsmv2", "aws-sdk-glue", "aws-sdk-migrationhub", "aws-sdk-mobile", "aws-sdk-pricing", "aws-sdk-costexplorer", "aws-sdk-mediaconvert", "aws-sdk-medialive", "aws-sdk-mediapackage", "aws-sdk-mediastore", "aws-sdk-mediastoredata", "aws-sdk-appsync", "aws-sdk-guardduty", "aws-sdk-mq", "aws-sdk-comprehend", "aws-sdk-iotjobsdataplane", "aws-sdk-kinesisvideo", "aws-sdk-kinesisvideoarchivedmedia", "aws-sdk-kinesisvideomedia", "aws-sdk-sagemaker", "aws-sdk-sagemakerruntime", "aws-sdk-translate", "aws-sdk-resourcegroups", "aws-sdk-alexaforbusiness", "aws-sdk-cloud9", "aws-sdk-serverlessapplicationrepository", "aws-sdk-servicediscovery", "aws-sdk-workmail", "aws-sdk-autoscalingplans", "aws-sdk-transcribeservice", "aws-sdk-connect", "aws-sdk-acmpca", "aws-sdk-fms", "aws-sdk-secretsmanager", "aws-sdk-iotanalytics", "aws-sdk-iot1clickdevicesservice", "aws-sdk-iot1clickprojects", "aws-sdk-pi", "aws-sdk-neptune", "aws-sdk-mediatailor", "aws-sdk-eks", "aws-sdk-macie", "aws-sdk-dlm", "aws-sdk-signer", "aws-sdk-chime", "aws-sdk-pinpointemail", "aws-sdk-pinpointsmsvoice", "aws-sdk-ram", "aws-sdk-route53resolver", "aws-sdk-s3control", "aws-sdk-quicksight", "aws-sdk-rdsdataservice", "aws-sdk-amplify", "aws-sdk-datasync", "aws-sdk-robomaker", "aws-sdk-transfer", "aws-sdk-globalaccelerator", "aws-sdk-comprehendmedical", "aws-sdk-kinesisanalyticsv2", "aws-sdk-mediaconnect", "aws-sdk-fsx", "aws-sdk-securityhub", "aws-sdk-appmesh", "aws-sdk-licensemanager", "aws-sdk-kafka", "aws-sdk-apigatewaymanagementapi", "aws-sdk-apigatewayv2", "aws-sdk-docdb", "aws-sdk-backup", "aws-sdk-worklink", "aws-sdk-textract"]
Looking up gems ["highline", "rubyforge", "rcov", "gemcutter", "allison", "rdoc", "safe_yaml", "configuration", "spoon", "addressable", "ffi", "RubyInline", "i18n", "memcache-client", "tzinfo", "minitest", "thread_safe", "concurrent-ruby", "method_source", "zeitwerk", "archive-tar-minitar", "mime-types-data", "cucumber", "diff-lcs", "spicycode-rcov", "syntax", "rspec-core", "rspec-expectations", "rspec-mocks", "spruz"]
Looking up gems ["net-scp", "psych", "public_suffix", "ZenTest", "ruby_parser", "minitar", "minitar-cli", "bones", "atomic", "rspec-support", "functional-ruby", "ref", "termios", "polyglot", "term-ansicolor", "treetop", "gherkin", "multi_test", "cucumber-core", "gherkin3", "cucumber-wire", "event-bus", "cucumber-expressions", "cucumber-formatter-dots"]
Looking up gems ["ParseTree", "sexp_processor", "little-plugger", "loquacious", "bones-extras", "bones-git", "net-ssh", "facets", "backports", "cucumber-tag_expressions", "tins", "powerbar", "jar-dependencies", "trollop", "cucumber-messages", "c21e", "os"]
Looking up gems ["SexpProcessor", "ansi", "hashie", "ruby-maven", "google-protobuf", "git", "bones-rcov", "bones-rspec", "bones-rubyforge", "bones-zentest", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium", "curses"]
Looking up gems ["thor", "maven-tools", "ruby-maven-libs"]
Looking up gems ["virtus"]
Looking up gems ["descendants_tracker", "axiom-types", "coercible", "equalizer"]
Looking up gems ["ice_nine", "adamantium"]
Looking up gems ["memoizable"]
Fetching version metadata from https://rubygems.org/
Looking up gems ["aws-sdk", "aws-sdk-core", "aws-sigv4", "aws-eventstream", "aws-sdk-kms", "builder", "multi_json", "multi_xml", "jamespath", "jmespath", "aws-partitions", "http-2", "httparty", "json", "nokogiri", "uuidtools", "aws-sdk-v1", "aws-sdk-resources", "aws-sdk-acm", "aws-sdk-apigateway", "aws-sdk-applicationautoscaling", "aws-sdk-applicationdiscoveryservice", "aws-sdk-autoscaling", "aws-sdk-budgets", "aws-sdk-cloudformation", "aws-sdk-cloudfront", "aws-sdk-cloudhsm", "aws-sdk-cloudsearch", "aws-sdk-cloudsearchdomain", "aws-sdk-cloudtrail", "aws-sdk-cloudwatch", "aws-sdk-cloudwatchevents", "aws-sdk-cloudwatchlogs", "aws-sdk-codecommit", "aws-sdk-codedeploy", "aws-sdk-codepipeline", "aws-sdk-cognitoidentity", "aws-sdk-cognitoidentityprovider", "aws-sdk-cognitosync", "aws-sdk-configservice", "aws-sdk-databasemigrationservice", "aws-sdk-datapipeline", "aws-sdk-devicefarm", "aws-sdk-directconnect", "aws-sdk-directoryservice", "aws-sdk-dynamodb", "aws-sdk-dynamodbstreams", "aws-sdk-ec2", "aws-sdk-ecr", "aws-sdk-ecs", "aws-sdk-efs", "aws-sdk-elasticache", "aws-sdk-elasticbeanstalk", "aws-sdk-elasticloadbalancing", "aws-sdk-elasticloadbalancingv2", "aws-sdk-elasticsearchservice", "aws-sdk-elastictranscoder", "aws-sdk-emr", "aws-sdk-firehose", "aws-sdk-gamelift", "aws-sdk-glacier", "aws-sdk-iam", "aws-sdk-importexport", "aws-sdk-inspector", "aws-sdk-iot", "aws-sdk-iotdataplane", "aws-sdk-kinesis", "aws-sdk-kinesisanalytics", "aws-sdk-lambda", "aws-sdk-lambdapreview", "aws-sdk-machinelearning", "aws-sdk-marketplacecommerceanalytics", "aws-sdk-marketplacemetering", "aws-sdk-opsworks", "aws-sdk-rds", "aws-sdk-redshift", "aws-sdk-route53", "aws-sdk-route53domains", "aws-sdk-s3", "aws-sdk-servicecatalog", "aws-sdk-ses", "aws-sdk-simpledb", "aws-sdk-sms", "aws-sdk-snowball", "aws-sdk-sns", "aws-sdk-sqs", "aws-sdk-ssm", "aws-sdk-storagegateway", "aws-sdk-support", "aws-sdk-swf", "aws-sdk-waf", "aws-sdk-workspaces", "aws-sdk-appstream", "aws-sdk-codebuild", "aws-sdk-health", "aws-sdk-lightsail", "aws-sdk-opsworkscm", "aws-sdk-pinpoint", "aws-sdk-polly", "aws-sdk-rekognition", "aws-sdk-sfn", "aws-sdk-shield", "aws-sdk-wafregional", "aws-sdk-xray", "aws-sdk-batch", "aws-sdk-clouddirectory", "aws-sdk-costandusagereportservice", "aws-sdk-lexruntimeservice", "aws-sdk-mturk", "aws-sdk-organizations", "aws-sdk-workdocs", "aws-sdk-codestar", "aws-sdk-lexmodelbuildingservice", "aws-sdk-resourcegroupstaggingapi", "aws-sdk-lex", "aws-sdk-marketplaceentitlementservice", "aws-sdk-states", "aws-sdk-athena", "echoe", "jnunemaker-crack", "crack", "activesupport", "hoe", "mime-types", "json_pure", "aws-sigv2", "launchy", "rake", "rspec", "racc", "rake-compiler", "rexical", "tenderlove-frex", "weakling", "mini_portile", "mini_portile2", "pkg-config", "aws-sdk-dax", "aws-sdk-greengrass", "aws-sdk-cloudhsmv2", "aws-sdk-glue", "aws-sdk-migrationhub", "aws-sdk-mobile", "aws-sdk-pricing", "aws-sdk-costexplorer", "aws-sdk-mediaconvert", "aws-sdk-medialive", "aws-sdk-mediapackage", "aws-sdk-mediastore", "aws-sdk-mediastoredata", "aws-sdk-appsync", "aws-sdk-guardduty", "aws-sdk-mq", "aws-sdk-comprehend", "aws-sdk-iotjobsdataplane", "aws-sdk-kinesisvideo", "aws-sdk-kinesisvideoarchivedmedia", "aws-sdk-kinesisvideomedia", "aws-sdk-sagemaker", "aws-sdk-sagemakerruntime", "aws-sdk-translate", "aws-sdk-resourcegroups", "aws-sdk-alexaforbusiness", "aws-sdk-cloud9", "aws-sdk-serverlessapplicationrepository", "aws-sdk-servicediscovery", "aws-sdk-workmail", "aws-sdk-autoscalingplans", "aws-sdk-transcribeservice", "aws-sdk-connect", "aws-sdk-acmpca", "aws-sdk-fms", "aws-sdk-secretsmanager", "aws-sdk-iotanalytics", "aws-sdk-iot1clickdevicesservice", "aws-sdk-iot1clickprojects", "aws-sdk-pi", "aws-sdk-neptune", "aws-sdk-mediatailor", "aws-sdk-eks", "aws-sdk-macie", "aws-sdk-dlm", "aws-sdk-signer", "aws-sdk-chime", "aws-sdk-pinpointemail", "aws-sdk-pinpointsmsvoice", "aws-sdk-ram", "aws-sdk-route53resolver", "aws-sdk-s3control", "aws-sdk-quicksight", "aws-sdk-rdsdataservice", "aws-sdk-amplify", "aws-sdk-datasync", "aws-sdk-robomaker", "aws-sdk-transfer", "aws-sdk-globalaccelerator", "aws-sdk-comprehendmedical", "aws-sdk-kinesisanalyticsv2", "aws-sdk-mediaconnect", "aws-sdk-fsx", "aws-sdk-securityhub", "aws-sdk-appmesh", "aws-sdk-licensemanager", "aws-sdk-kafka", "aws-sdk-apigatewaymanagementapi", "aws-sdk-apigatewayv2", "aws-sdk-docdb", "aws-sdk-backup", "aws-sdk-worklink", "aws-sdk-textract", "highline", "rubyforge", "rcov", "gemcutter", "allison", "rdoc", "safe_yaml", "configuration", "spoon", "addressable", "ffi", "RubyInline", "i18n", "memcache-client", "tzinfo", "minitest", "thread_safe", "concurrent-ruby", "method_source", "zeitwerk", "archive-tar-minitar", "mime-types-data", "cucumber", "diff-lcs", "spicycode-rcov", "syntax", "rspec-core", "rspec-expectations", "rspec-mocks", "spruz", "net-scp", "psych", "public_suffix", "ZenTest", "ruby_parser", "minitar", "minitar-cli", "bones", "atomic", "rspec-support", "functional-ruby", "ref", "termios", "polyglot", "term-ansicolor", "treetop", "gherkin", "multi_test", "cucumber-core", "gherkin3", "cucumber-wire", "event-bus", "cucumber-expressions", "cucumber-formatter-dots", "ParseTree", "sexp_processor", "little-plugger", "loquacious", "bones-extras", "bones-git", "net-ssh", "facets", "backports", "cucumber-tag_expressions", "tins", "powerbar", "jar-dependencies", "trollop", "cucumber-messages", "c21e", "os", "SexpProcessor", "ansi", "hashie", "ruby-maven", "google-protobuf", "git", "bones-rcov", "bones-rspec", "bones-rubyforge", "bones-zentest", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium", "curses", "thor", "maven-tools", "ruby-maven-libs", "virtus", "descendants_tracker", "axiom-types", "coercible", "equalizer", "ice_nine", "adamantium", "memoizable"]
Starting resolution (2019-03-15 22:47:58 +0000)
Resolving dependencies...Creating possibility state for aws-sdk-core (~> 3.44.0) (3 remaining)
  Attempting to activate aws-sdk-core (3.44.2)
  Activated aws-sdk-core at aws-sdk-core (3.44.2)
  Requiring nested dependencies (aws-eventstream (~> 1.0), aws-partitions (~> 1.0), aws-sigv4 (~> 1.0), jmespath (~> 1.0))
  Creating possibility state for aws-partitions (~> 1.123.0) (1 remaining)
    Attempting to activate aws-partitions (1.123.0)
    Activated aws-partitions at aws-partitions (1.123.0)
    Requiring nested dependencies ()
    Creating possibility state for aws-partitions (~> 1.0) (144 remaining)
      Attempting to activate aws-partitions (1.144.0)
      Found existing spec (aws-partitions (1.123.0))
      Creating possibility state for aws-sdk-ec2 (~> 1.63.0) (1 remaining)
        Attempting to activate aws-sdk-ec2 (1.63.0)
        Activated aws-sdk-ec2 at aws-sdk-ec2 (1.63.0)
        Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
        Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
          Attempting to activate aws-sdk-core (3.47.0)
          Found existing spec (aws-sdk-core (3.44.2))
          Creating possibility state for aws-sdk-ssm (~> 1.34.0) (1 remaining)
            Attempting to activate aws-sdk-ssm (1.34.0)
            Activated aws-sdk-ssm at aws-sdk-ssm (1.34.0)
            Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
            Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
              Attempting to activate aws-sdk-core (3.47.0)
              Found existing spec (aws-sdk-core (3.44.2))
              Creating possibility state for aws-sdk-route53 (~> 1.16.0) (1 remaining)
                Attempting to activate aws-sdk-route53 (1.16.0)
                Activated aws-sdk-route53 at aws-sdk-route53 (1.16.0)
                Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
                Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
                  Attempting to activate aws-sdk-core (3.47.0)
                  Found existing spec (aws-sdk-core (3.44.2))
                  Creating possibility state for aws-sdk-cloudformation (~> 1.13.0) (1 remaining)
                    Attempting to activate aws-sdk-cloudformation (1.13.0)
                    Activated aws-sdk-cloudformation at aws-sdk-cloudformation (1.13.0)
                    Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
                    Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
                      Attempting to activate aws-sdk-core (3.47.0)
                      Found existing spec (aws-sdk-core (3.44.2))
                      Creating possibility state for aws-sdk-cloudwatch (~> 1.13.0) (1 remaining)
                        Attempting to activate aws-sdk-cloudwatch (1.13.0)
                        Activated aws-sdk-cloudwatch at aws-sdk-cloudwatch (1.13.0)
                        Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
                        Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
                          Attempting to activate aws-sdk-core (3.47.0)
                          Found existing spec (aws-sdk-core (3.44.2))
                          Creating possibility state for aws-sdk-iam (~> 1.13.0) (1 remaining)
                            Attempting to activate aws-sdk-iam (1.13.0)
                            Activated aws-sdk-iam at aws-sdk-iam (1.13.0)
                            Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
                            Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
                              Attempting to activate aws-sdk-core (3.47.0)
                              Found existing spec (aws-sdk-core (3.44.2))
                              Creating possibility state for aws-sdk-kms (~> 1.13.0) (1 remaining)
                                Attempting to activate aws-sdk-kms (1.13.0)
                                Activated aws-sdk-kms at aws-sdk-kms (1.13.0)
                                Requiring nested dependencies (aws-sdk-core (>= 3.39.0, ~> 3), aws-sigv4 (~> 1.0))
                                Creating possibility state for aws-sdk-core (>= 3.39.0, ~> 3) (13 remaining)
                                  Attempting to activate aws-sdk-core (3.47.0)
                                  Found existing spec (aws-sdk-core (3.44.2))
                                  Creating possibility state for aws-sdk-s3 (~> 1.30.0) (2 remaining)
                                    Attempting to activate aws-sdk-s3 (1.30.1)
                                    Activated aws-sdk-s3 at aws-sdk-s3 (1.30.1)...................

guidance third-party

Source

Paul1987

Most helpful comment

Best thing to do here would be to require the AWS service gems you need, rather than all ~150 services.

For example, if you use S3 and EC2, you could simply install aws-sdk-s3 and aws-sdk-ec2 and the number of gems you install will be cut drastically.

awood45 on 18 Mar 2019

👍3

All 3 comments

Best thing to do here would be to require the AWS service gems you need, rather than all ~150 services.

For example, if you use S3 and EC2, you could simply install aws-sdk-s3 and aws-sdk-ec2 and the number of gems you install will be cut drastically.

awood45 on 18 Mar 2019

👍3

Thanks for the suggestion @awood45 - This fixed our problem!