Serving: TFS doesn't handle grpc request deadline/timeout properly

Created on 20 Feb 2019 · 30Comments · Source: tensorflow/serving

Bug Report

System information

Linux 4.4.121-k8s #1 SMP Sun Mar 11 19:39:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux:
On all 1.3, 1.7 and 1.8 i can observe same behaviours:

Describe the problem

Even i set the deadline/timeout for all the gRPC as 30s, tensorflow serving doesn't respect that, still queue/run all the requests until it reaches its end;

Thus in my go grpc client i can observe that all requests already returned with panic: rpc error: code = DeadlineExceeded desc = context deadline exceeded, but in TFS model server i still see incoming request for a long time, and the request duration(processing time) keep increasing from the normal around 30s to 1m, 2m, 3m ... until it finishes all requests.

What i expect is the TFS model server ends the request and releases the resources immediately when the request already exceeds its deadline or the client was already gone.

Exact Steps to Reproduce

Just use whatever grpc client to send heavy request to TFS model server continuously, like 1 request per second, and last 1 minute like that, and for each request set a short deadline/timeout, then observe whether TFS model server respect the deadline/timeout.

bug performance

Source

lnshi

All 30 comments

Could you please provide some more information like

TensorFlow Serving installed from (source or binary) :
TensorFlow Serving version :
Docker version :

Harshini-Gadige on 20 Feb 2019

@hgadig we build from source, and for version 1.3, 1,7 and 1.8 all behaves same.

lnshi on 21 Feb 2019

@hgadig Actually, after then i runned with tensorflow serving official docker image: https://hub.docker.com/r/tensorflow/serving/tags?page=1: tensorflow/serving:1.11.1, same behaviours observed: tfs doesn't end the request and release the occupied resources immediately upon deadline exceed or client is gone.

lnshi on 21 Feb 2019

same issue, waiting for help.

songboyu on 31 Mar 2019

@minglotus-6 any update or in development activities?

Really serious issue, same on tensorflow/serving:1.12.0, all the requests will jam in the model server until it got forcedly killed, no resources were released properly before that...

lnshi on 16 Apr 2019

@netfs do you have any idea about this?
tensorflow::RunOptions is correctly set here
propagating enforce_session_run_timeout_
originally set to inherit server_options.enforce_session_run_timeout. But looking in modelserver's main.cc I don't see a flag setting tensorflow::serving::main::Server::Options.enforce_session_run_timeout (which presumably defaults to false?)

Am I missing something or is there a reason why the flag has been omitted?

misterpeddy on 17 Apr 2019

enforce_session_run_timeout is set to true by default:

https://github.com/tensorflow/serving/blob/96a716ca31f753b0c3efc1ef60779b77f5c60845/tensorflow_serving/model_servers/server.h#L79

netfs on 17 Apr 2019

@netfs hmm, what could cause it doesn't get respected then?

lnshi on 18 Apr 2019

Hey lnshi,
Thanks for reporting this.

The current investigation result is that, server propagates client-specified timeout to tensorflow sessions.

The most useful piece of information for us to continue investigation would be a stack trace for those threads on servers that continue to be active (when the RPC is cancelled on client side, user gets an deadline exceeded error).

Could you help provide the model, request and response when you encounter this issue, so we can proceed to see the stack traces? Appreciate it!

minglotus-6 on 10 May 2019

@minglotus-6 thanks a lot for you time.

Firstly according to my experiments looks this issue is NOT specific to any model, i can easily reproduce it with TFS public pre-trained object detection model, inception model version 1, 2, and 3, and other two ourself build TFS models.

Just now i tried and was able to reproduce it with this latest inception model: faster_rcnn_inception_resnet_v2_atrous_oidv4

model_config_list.conf

model_config_list: {
  config: {
    name: "inception",
    base_path: "/models/inception",
    model_platform: "tensorflow",
    model_version_policy: {
      specific: {
        versions: [1, 2, 3]
      }
    }
  }
}

Simple python script to send the gRPC inference request(input is just a small 2KB jpg image):

import grpc
from grpc._cython.cygrpc import CompressionAlgorithm
from grpc._cython.cygrpc import CompressionLevel

from grpc.beta import implementations
import tensorflow as tf
import os, time
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2

import numpy as np

MODEL_NAME = 'inception'
MODEL_SERVER_HOST = 'localhost'
MODEL_SERVER_PORT = 9000

opts = [('grpc.default_compression_algorithm', CompressionAlgorithm.gzip),
            ('grpc.grpc.default_compression_level', CompressionLevel.high)]

ch = grpc.insecure_channel(MODEL_SERVER_HOST + ":" + str(MODEL_SERVER_PORT), options=opts)
channel = implementations.Channel(ch)
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

request = predict_pb2.PredictRequest()
request.model_spec.name = MODEL_NAME
request.model_spec.version.value = 3
request.model_spec.signature_name = 'serving_default'

test = [[[[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [242, 255, 255], [248, 255, 255], [255, 253, 255], [255, 241, 235], [255, 244, 232], [254, 198, 181], [63, 0, 0], [150, 87, 70], [227, 171, 158], [255, 228, 219], [255, 246, 243], [255, 243, 243], [255, 250, 254], [249, 255, 255], [243, 255, 255], [239, 255, 255], [252, 252, 254], [254, 255, 255], [252, 255, 255], [251, 255, 255], [248, 255, 255], [247, 255, 255], [241, 255, 253], [238, 253, 250], [245, 255, 255], [249, 255, 255], [252, 255, 255], [255, 254, 255], [252, 246, 250], [255, 250, 255], [255, 246, 254], [255, 238, 247], [255, 232, 239], [255, 241, 234], [230, 183, 155], [151, 106, 65], [63, 19, 0], [219, 178, 156], [255, 244, 248], [255, 241, 255], [255, 247, 255], [255, 251, 255], [245, 249, 250], [250, 255, 243], [251, 255, 241], [252, 255, 246], [255, 254, 255], [255, 249, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [242, 255, 255], [249, 255, 255], [248, 239, 240], [255, 244, 237], [244, 202, 188], [62, 7, 0], [233, 171, 148], [202, 140, 119], [93, 36, 19], [174, 128, 115], [255, 223, 216], [255, 243, 242], [255, 252, 252], [252, 255, 255], [248, 255, 255], [245, 255, 255], [255, 255, 255], [255, 255, 255], [253, 254, 255], [247, 253, 253], [246, 255, 254], [247, 255, 255], [247, 255, 255], [247, 255, 255], [244, 254, 253], [242, 251, 250], [249, 253, 254], [251, 249, 252], [255, 252, 255], [251, 238, 245], [255, 249, 255], [255, 248, 255], [255, 238, 240], [164, 115, 101], [78, 29, 0], [192, 145, 101], [205, 162, 119], [73, 31, 7], [215, 179, 179], [255, 244, 255], [253, 235, 255], [255, 251, 255], [254, 255, 255], [251, 255, 244], [251, 255, 241], [251, 254, 245], [255, 254, 255], [251, 247, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [247, 255, 255], [250, 253, 255], [255, 252, 250], [255, 248, 237], [110, 67, 48], [182, 126, 99], [255, 196, 165], [255, 199, 171], [240, 183, 156], [121, 73, 53], [102, 66, 52], [255, 241, 233], [255, 252, 248], [237, 236, 234], [252, 255, 255], [251, 255, 255], [251, 249, 252], [254, 252, 255], [255, 255, 255], [252, 255, 255], [251, 255, 255], [249, 255, 255], [246, 255, 255], [244, 254, 253], [249, 255, 255], [251, 255, 255], [254, 255, 255], [248, 243, 247], [251, 242, 247], [255, 250, 255], [252, 235, 243], [253, 233, 242], [139, 86, 78], [126, 74, 53], [239, 190, 150], [249, 201, 152], [255, 214, 165], [191, 148, 116], [80, 41, 34], [251, 221, 233], [255, 242, 255], [255, 251, 255], [249, 251, 248], [252, 255, 243], [252, 255, 241], [249, 252, 243], [254, 253, 255], [254, 250, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [243, 250, 255], [250, 249, 254], [255, 252, 246], [156, 127, 111], [128, 82, 56], [255, 218, 183], [240, 177, 136], [246, 181, 141], [255, 200, 164], [255, 210, 179], [157, 116, 94], [81, 52, 38], [239, 221, 211], [255, 252, 248], [255, 254, 251], [255, 253, 254], [255, 253, 253], [255, 252, 252], [254, 250, 249], [250, 250, 248], [249, 251, 248], [250, 255, 251], [251, 255, 253], [251, 255, 253], [252, 255, 253], [241, 243, 240], [253, 252, 250], [255, 252, 253], [255, 248, 250], [253, 237, 240], [255, 248, 253], [109, 88, 93], [153, 98, 78], [255, 202, 171], [242, 190, 142], [233, 182, 125], [231, 182, 126], [245, 200, 161], [140, 98, 82], [159, 127, 130], [255, 247, 255], [255, 251, 255], [241, 240, 235], [252, 255, 243], [252, 255, 243], [247, 253, 243], [250, 249, 254], [255, 253, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [253, 253, 255], [252, 248, 249], [246, 229, 219], [63, 31, 8], [232, 186, 150], [229, 171, 125], [247, 181, 129], [255, 189, 137], [254, 192, 143], [224, 171, 129], [246, 201, 168], [164, 130, 105], [123, 95, 81], [251, 231, 222], [255, 250, 246], [253, 238, 235], [255, 252, 248], [255, 252, 248], [255, 253, 248], [255, 254, 248], [255, 254, 248], [255, 255, 248], [253, 253, 245], [251, 251, 243], [255, 255, 248], [255, 253, 247], [249, 240, 235], [255, 246, 243], [255, 250, 248], [255, 244, 243], [135, 111, 111], [122, 96, 97], [253, 197, 162], [236, 182, 136], [244, 189, 132], [239, 184, 119], [247, 193, 131], [255, 209, 162], [209, 165, 138], [67, 30, 22], [232, 208, 208], [255, 252, 250], [255, 252, 243], [251, 255, 240], [252, 255, 243], [249, 254, 247], [253, 254, 255], [255, 253, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 253, 255], [255, 252, 250], [175, 153, 139], [128, 93, 65], [244, 195, 152], [252, 193, 137], [246, 180, 119], [255, 203, 140], [243, 181, 120], [253, 196, 143], [251, 202, 159], [237, 196, 164], [110, 75, 53], [171, 142, 126], [255, 244, 233], [255, 248, 241], [255, 246, 235], [255, 245, 234], [255, 245, 233], [255, 247, 234], [255, 249, 236], [255, 253, 239], [255, 253, 239], [255, 253, 239], [247, 235, 221], [255, 252, 239], [255, 251, 239], [255, 243, 232], [255, 249, 239], [190, 161, 153], [66, 36, 28], [236, 203, 196], [255, 199, 149], [243, 187, 130], [255, 213, 146], [243, 185, 112], [241, 184, 115], [244, 189, 133], [255, 216, 178], [147, 109, 88], [149, 121, 110], [253, 236, 228], [255, 255, 243], [250, 252, 238], [247, 251, 237], [252, 255, 250], [254, 255, 255], [254, 254, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [247, 237, 238], [255, 241, 234], [96, 69, 50], [204, 166, 130], [244, 192, 142], [248, 187, 124], [255, 206, 135], [230, 162, 89], [234, 170, 99], [255, 201, 138], [249, 198, 145], [226, 178, 138], [222, 178, 149], [89, 48, 28], [212, 175, 159], [255, 245, 232], [251, 220, 199], [232, 201, 180], [200, 172, 150], [170, 144, 121], [151, 125, 102], [145, 121, 97], [147, 125, 101], [152, 130, 106], [192, 168, 144], [201, 175, 152], [229, 201, 179], [255, 238, 217], [245, 210, 190], [67, 29, 10], [228, 187, 169], [222, 181, 163], [234, 176, 112], [236, 177, 109], [238, 178, 105], [242, 180, 103], [255, 200, 127], [253, 195, 132], [234, 182, 134], [235, 192, 158], [102, 71, 50], [229, 210, 195], [255, 254, 241], [251, 251, 239], [249, 252, 243], [252, 255, 251], [252, 255, 255], [251, 255, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 251, 248], [216, 196, 185], [92, 62, 38], [240, 198, 158], [247, 195, 138], [250, 190, 120], [255, 188, 110], [240, 173, 92], [244, 178, 100], [225, 165, 95], [240, 186, 126], [255, 218, 169], [237, 189, 151], [179, 132, 104], [74, 28, 5], [152, 105, 85], [96, 53, 21], [115, 72, 40], [142, 101, 69], [169, 130, 97], [186, 147, 114], [188, 152, 118], [182, 146, 112], [177, 141, 107], [146, 110, 76], [133, 94, 61], [90, 49, 17], [149, 106, 74], [86, 41, 10], [149, 102, 72], [240, 191, 161], [255, 204, 175], [253, 197, 122], [244, 187, 110], [229, 168, 88], [226, 163, 84], [255, 193, 116], [239, 178, 111], [250, 195, 138], [243, 196, 152], [107, 72, 40], [209, 187, 164], [250, 238, 222], [253, 251, 239], [255, 255, 250], [249, 254, 250], [245, 249, 248], [252, 255, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 250, 243], [171, 145, 130], [152, 116, 90], [255, 210, 168], [244, 190, 130], [241, 179, 106], [251, 184, 105], [236, 170, 86], [234, 169, 89], [237, 177, 104], [234, 179, 114], [202, 149, 95], [151, 98, 54], [126, 74, 37], [158, 105, 74], [207, 153, 125], [245, 192, 150], [246, 193, 151], [247, 196, 153], [250, 199, 156], [249, 200, 157], [249, 200, 157], [248, 199, 156], [248, 199, 156], [254, 205, 162], [242, 193, 150], [237, 186, 143], [221, 168, 126], [174, 119, 78], [132, 74, 34], [149, 91, 51], [198, 139, 99], [242, 187, 104], [220, 164, 81], [246, 185, 104], [243, 177, 99], [232, 166, 90], [251, 187, 116], [249, 191, 127], [255, 209, 156], [188, 151, 109], [136, 110, 83], [255, 250, 235], [255, 254, 248], [253, 253, 251], [254, 255, 255], [248, 252, 251], [252, 255, 253]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 240, 229], [113, 82, 64], [213, 174, 145], [230, 181, 138], [246, 190, 131], [255, 206, 133], [245, 180, 100], [234, 169, 87], [233, 170, 90], [242, 182, 109], [218, 161, 94], [174, 119, 62], [180, 127, 77], [235, 178, 135], [255, 205, 167], [253, 193, 157], [246, 184, 133], [247, 185, 134], [247, 187, 135], [249, 189, 137], [250, 190, 138], [248, 191, 138], [247, 190, 137], [246, 189, 136], [246, 189, 136], [242, 182, 130], [248, 188, 136], [255, 202, 151], [255, 200, 149], [234, 171, 120], [199, 136, 85], [179, 116, 65], [230, 178, 92], [238, 183, 100], [237, 176, 96], [232, 166, 92], [255, 189, 116], [253, 186, 115], [255, 199, 132], [239, 187, 127], [215, 175, 126], [110, 81, 49], [254, 237, 221], [255, 252, 250], [248, 246, 249], [254, 255, 255], [246, 251, 247], [252, 255, 251]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [250, 218, 203], [95, 60, 38], [234, 191, 159], [244, 193, 148], [253, 194, 136], [233, 170, 101], [243, 177, 101], [237, 174, 95], [238, 176, 99], [246, 188, 115], [245, 190, 125], [240, 186, 126], [247, 192, 138], [255, 200, 152], [253, 192, 147], [233, 172, 127], [251, 186, 128], [252, 187, 129], [252, 187, 129], [253, 188, 130], [253, 188, 130], [253, 188, 130], [250, 188, 129], [249, 187, 128], [255, 196, 137], [255, 192, 134], [251, 186, 128], [250, 185, 127], [255, 195, 137], [255, 201, 143], [255, 189, 131], [235, 169, 111], [235, 188, 100], [245, 193, 110], [227, 169, 95], [232, 167, 99], [247, 178, 111], [242, 173, 104], [243, 180, 109], [255, 202, 136], [242, 201, 147], [89, 59, 23], [234, 215, 200], [255, 251, 253], [247, 242, 249], [254, 255, 255], [245, 250, 246], [252, 255, 248]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [231, 196, 176], [116, 79, 53], [231, 186, 153], [252, 199, 155], [251, 192, 136], [229, 168, 103], [243, 180, 109], [240, 178, 105], [249, 190, 120], [239, 184, 117], [246, 192, 128], [255, 208, 148], [252, 197, 141], [231, 174, 119], [235, 175, 121], [255, 197, 143], [255, 191, 130], [255, 191, 130], [255, 192, 130], [255, 192, 130], [255, 191, 129], [255, 191, 129], [254, 190, 128], [254, 190, 128], [246, 182, 120], [255, 193, 131], [255, 194, 132], [248, 184, 122], [245, 181, 119], [254, 190, 128], [255, 196, 134], [255, 195, 133], [243, 199, 112], [247, 198, 119], [239, 182, 115], [243, 179, 118], [224, 155, 96], [255, 193, 128], [238, 175, 104], [255, 201, 131], [251, 209, 151], [105, 73, 35], [214, 190, 178], [255, 250, 255], [253, 245, 255], [255, 254, 255], [247, 252, 248], [252, 255, 246]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [221, 183, 160], [127, 86, 58], [253, 207, 173], [233, 180, 136], [249, 189, 137], [255, 196, 137], [235, 174, 111], [240, 180, 117], [248, 193, 129], [246, 195, 132], [239, 189, 130], [230, 180, 121], [233, 181, 123], [249, 193, 134], [255, 200, 140], [255, 200, 138], [253, 192, 129], [252, 191, 128], [252, 191, 128], [251, 190, 127], [250, 189, 126], [249, 188, 125], [249, 188, 125], [249, 188, 125], [251, 190, 127], [249, 188, 125], [248, 187, 124], [251, 190, 127], [252, 192, 129], [252, 192, 129], [250, 190, 127], [248, 188, 125], [235, 194, 112], [253, 206, 134], [251, 197, 137], [248, 185, 134], [234, 164, 113], [255, 191, 130], [255, 206, 138], [228, 174, 102], [247, 202, 143], [143, 109, 72], [198, 171, 162], [255, 243, 253], [255, 250, 255], [251, 250, 255], [253, 255, 252], [252, 255, 244]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [213, 171, 146], [138, 94, 65], [255, 207, 171], [249, 194, 153], [239, 178, 131], [182, 121, 67], [225, 164, 109], [247, 188, 132], [243, 188, 131], [252, 201, 144], [252, 203, 147], [242, 193, 135], [242, 192, 131], [253, 199, 135], [249, 192, 125], [233, 174, 106], [249, 194, 130], [248, 193, 129], [247, 192, 128], [248, 190, 127], [247, 189, 126], [247, 189, 126], [248, 190, 127], [248, 190, 127], [255, 201, 138], [248, 190, 127], [241, 186, 122], [245, 190, 126], [245, 191, 127], [242, 188, 124], [245, 191, 127], [253, 199, 135], [233, 198, 117], [242, 199, 131], [238, 187, 134], [254, 193, 149], [250, 182, 137], [173, 104, 49], [255, 194, 126], [248, 191, 120], [244, 198, 138], [163, 127, 91], [188, 161, 152], [255, 242, 255], [255, 249, 255], [248, 245, 255], [254, 255, 251], [251, 255, 242]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [214, 170, 143], [147, 102, 71], [236, 186, 151], [255, 201, 160], [174, 116, 70], [131, 69, 20], [236, 176, 124], [250, 192, 142], [245, 192, 140], [239, 189, 138], [238, 191, 139], [241, 194, 138], [239, 191, 129], [235, 184, 118], [245, 191, 121], [255, 204, 131], [248, 196, 136], [247, 195, 135], [246, 194, 134], [245, 193, 133], [244, 192, 132], [247, 193, 133], [248, 194, 134], [249, 195, 135], [233, 181, 121], [240, 188, 128], [251, 199, 139], [255, 206, 145], [249, 199, 138], [237, 189, 127], [236, 188, 126], [241, 193, 131], [240, 206, 132], [230, 191, 126], [231, 180, 133], [253, 193, 156], [252, 184, 145], [151, 81, 30], [178, 113, 47], [249, 192, 121], [254, 208, 148], [148, 112, 76], [187, 158, 152], [255, 247, 255], [255, 249, 255], [251, 247, 255], [254, 255, 251], [249, 255, 240]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [224, 180, 151], [139, 92, 62], [238, 188, 153], [214, 159, 120], [88, 29, 0], [209, 148, 101], [255, 202, 155], [245, 189, 140], [246, 194, 146], [239, 191, 142], [236, 190, 140], [239, 194, 137], [246, 198, 136], [247, 196, 130], [246, 192, 120], [246, 190, 115], [243, 194, 135], [243, 194, 135], [241, 192, 133], [241, 191, 132], [241, 191, 132], [242, 192, 133], [244, 194, 135], [245, 195, 136], [246, 196, 137], [246, 196, 137], [243, 194, 135], [238, 192, 132], [239, 193, 133], [242, 198, 137], [242, 198, 137], [239, 195, 134], [236, 204, 131], [237, 199, 137], [243, 195, 149], [245, 185, 151], [252, 184, 149], [240, 172, 123], [107, 43, 0], [187, 130, 59], [255, 220, 160], [125, 87, 51], [187, 158, 152], [255, 247, 255], [253, 241, 255], [255, 252, 255], [254, 255, 251], [248, 255, 237]], [[249, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [251, 249, 250], [254, 252, 253], [255, 254, 255], [255, 254, 255], [255, 253, 254], [253, 251, 252], [255, 253, 254], [255, 255, 255], [229, 189, 187], [124, 84, 74], [234, 190, 165], [71, 24, 0], [208, 156, 98], [255, 205, 136], [250, 187, 116], [253, 184, 117], [255, 187, 128], [255, 185, 137], [255, 185, 145], [255, 187, 150], [252, 190, 151], [248, 191, 146], [243, 191, 141], [240, 190, 137], [249, 195, 109], [249, 194, 112], [249, 193, 119], [249, 191, 128], [249, 189, 135], [249, 188, 141], [249, 188, 141], [249, 189, 139], [245, 186, 130], [253, 195, 132], [255, 199, 130], [251, 195, 120], [250, 194, 117], [253, 197, 120], [252, 196, 119], [247, 191, 116], [252, 190, 129], [252, 190, 129], [255, 196, 133], [255, 191, 127], [241, 180, 117], [255, 198, 141], [220, 159, 112], [84, 28, 0], [235, 185, 162], [139, 95, 84], [207, 172, 170], [255, 247, 250], [255, 245, 248], [255, 248, 251], [255, 254, 255], [251, 253, 252]], [[251, 255, 255], [251, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [254, 252, 253], [255, 254, 255], [255, 254, 255], [255, 254, 255], [254, 252, 253], [253, 251, 252], [253, 253, 253], [255, 255, 255], [244, 204, 202], [116, 73, 64], [114, 70, 45], [171, 124, 82], [247, 197, 138], [242, 187, 120], [246, 186, 116], [255, 192, 126], [255, 197, 138], [255, 189, 140], [247, 180, 137], [244, 179, 139], [244, 185, 141], [245, 192, 142], [244, 193, 138], [240, 191, 132], [247, 195, 111], [247, 194, 114], [247, 193, 121], [247, 192, 128], [247, 190, 135], [247, 189, 141], [247, 189, 141], [247, 189, 139], [251, 195, 138], [250, 195, 131], [246, 192, 122], [242, 188, 114], [243, 189, 115], [246, 192, 118], [247, 193, 121], [245, 191, 119], [248, 189, 129], [245, 187, 124], [248, 188, 125], [253, 194, 128], [245, 185, 122], [255, 198, 141], [248, 190, 140], [171, 115, 78], [131, 81, 56], [99, 55, 42], [230, 197, 192], [255, 248, 248], [255, 248, 251], [255, 254, 255], [254, 255, 255], [251, 255, 255]], [[252, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [253, 253, 253], [252, 252, 252], [254, 254, 254], [255, 255, 255], [255, 239, 238], [137, 94, 85], [86, 42, 17], [230, 185, 143], [247, 198, 140], [236, 185, 119], [255, 199, 130], [250, 191, 125], [247, 185, 126], [252, 190, 139], [255, 195, 148], [253, 195, 149], [246, 190, 139], [239, 189, 130], [240, 192, 126], [242, 198, 127], [246, 195, 116], [246, 194, 119], [246, 193, 125], [246, 192, 132], [246, 191, 137], [246, 190, 141], [246, 190, 141], [246, 191, 137], [250, 195, 138], [244, 190, 126], [241, 188, 120], [244, 191, 121], [248, 195, 125], [249, 196, 128], [249, 195, 131], [251, 197, 133], [246, 194, 136], [244, 192, 132], [242, 188, 124], [253, 198, 133], [248, 191, 124], [245, 187, 124], [255, 200, 145], [240, 187, 143], [123, 74, 42], [108, 67, 49], [255, 238, 229], [255, 249, 246], [255, 246, 247], [253, 253, 255], [249, 255, 255], [247, 255, 255]], [[254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [254, 254, 254], [254, 254, 254], [254, 254, 254], [255, 255, 255], [255, 255, 255], [255, 241, 237], [134, 88, 75], [173, 127, 101], [238, 191, 149], [236, 189, 133], [240, 191, 125], [248, 197, 131], [244, 190, 126], [238, 183, 126], [243, 188, 134], [249, 193, 144], [248, 195, 143], [244, 194, 135], [241, 194, 126], [239, 196, 118], [241, 199, 117], [243, 195, 119], [243, 195, 123], [243, 194, 128], [243, 193, 134], [243, 192, 137], [243, 192, 139], [243, 192, 139], [243, 192, 135], [243, 193, 132], [240, 191, 125], [242, 193, 126], [249, 200, 133], [249, 200, 134], [244, 194, 133], [242, 191, 134], [245, 194, 139], [238, 193, 138], [244, 198, 139], [239, 191, 127], [249, 198, 132], [248, 195, 127], [234, 179, 114], [247, 191, 132], [255, 202, 152], [194, 146, 108], [125, 85, 60], [255, 241, 227], [255, 250, 243], [254, 249, 246], [251, 255, 255], [245, 255, 255], [238, 255, 255]], [[255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [254, 254, 254], [254, 254, 254], [253, 253, 253], [254, 254, 254], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 205, 197], [106, 55, 38], [233, 184, 154], [244, 197, 153], [253, 206, 150], [237, 192, 127], [228, 180, 114], [246, 198, 134], [250, 201, 143], [240, 190, 139], [231, 181, 130], [232, 182, 129], [241, 193, 131], [247, 200, 128], [244, 199, 116], [239, 195, 106], [242, 195, 125], [242, 195, 127], [242, 194, 130], [242, 193, 134], [242, 193, 137], [242, 193, 137], [242, 193, 135], [242, 194, 132], [242, 194, 128], [242, 195, 127], [245, 198, 130], [246, 198, 132], [242, 193, 134], [236, 186, 133], [236, 185, 138], [240, 189, 144], [226, 186, 134], [242, 201, 145], [233, 189, 128], [239, 191, 125], [249, 196, 126], [241, 187, 117], [247, 192, 127], [248, 196, 139], [227, 178, 135], [111, 70, 40], [216, 188, 167], [255, 251, 241], [255, 255, 250], [249, 255, 255], [242, 255, 255], [231, 252, 253]], [[255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [253, 253, 253], [251, 253, 252], [251, 253, 252], [253, 255, 254], [254, 255, 255], [254, 255, 255], [254, 255, 255], [252, 254, 253], [205, 148, 129], [145, 89, 64], [242, 188, 152], [240, 190, 141], [250, 201, 142], [243, 198, 131], [236, 191, 126], [244, 198, 138], [238, 191, 137], [243, 195, 147], [249, 201, 153], [251, 201, 150], [246, 198, 136], [241, 194, 122], [241, 195, 110], [243, 198, 107], [242, 195, 127], [242, 194, 128], [242, 194, 132], [242, 193, 134], [242, 193, 135], [242, 193, 135], [242, 194, 132], [242, 194, 128], [237, 190, 120], [242, 195, 125], [243, 196, 126], [240, 192, 128], [240, 191, 133], [245, 194, 147], [249, 197, 157], [250, 198, 161], [228, 192, 144], [247, 207, 156], [234, 192, 134], [235, 187, 121], [245, 192, 122], [249, 193, 119], [253, 196, 127], [241, 185, 124], [253, 201, 153], [141, 98, 63], [163, 131, 108], [255, 251, 239], [255, 255, 250], [241, 250, 249], [242, 255, 255], [238, 255, 255]], [[255, 252, 255], [255, 253, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [253, 255, 254], [252, 254, 253], [252, 254, 253], [254, 255, 255], [254, 255, 255], [254, 254, 254], [250, 250, 250], [246, 246, 246], [161, 98, 67], [215, 152, 117], [253, 195, 149], [231, 176, 119], [239, 188, 123], [249, 200, 133], [250, 202, 138], [240, 194, 135], [231, 183, 134], [237, 188, 145], [244, 192, 152], [244, 193, 148], [241, 190, 135], [240, 189, 123], [242, 193, 116], [248, 197, 114], [243, 194, 128], [243, 193, 130], [243, 193, 132], [243, 193, 132], [243, 193, 134], [243, 193, 132], [243, 194, 128], [243, 194, 125], [235, 187, 113], [243, 195, 121], [244, 196, 124], [239, 190, 124], [244, 193, 138], [253, 200, 156], [247, 193, 157], [231, 176, 146], [223, 184, 143], [242, 201, 155], [240, 195, 140], [242, 192, 129], [244, 190, 120], [248, 188, 115], [255, 196, 124], [240, 180, 117], [255, 212, 158], [212, 164, 126], [123, 87, 61], [251, 229, 215], [255, 254, 248], [242, 248, 248], [243, 255, 255], [240, 255, 255]], [[255, 251, 255], [255, 252, 255], [255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [253, 255, 254], [254, 255, 255], [255, 255, 255], [253, 253, 253], [247, 245, 246], [241, 239, 240], [125, 58, 16], [225, 161, 113], [255, 206, 152], [242, 184, 120], [255, 204, 134], [244, 191, 121], [235, 186, 120], [245, 194, 139], [255, 210, 165], [224, 170, 134], [187, 132, 101], [189, 133, 98], [223, 166, 123], [252, 195, 140], [253, 198, 133], [244, 187, 116], [246, 192, 128], [246, 192, 130], [246, 192, 130], [246, 192, 132], [246, 192, 132], [246, 192, 128], [246, 193, 125], [246, 194, 121], [242, 190, 115], [252, 200, 125], [251, 199, 126], [243, 189, 125], [245, 190, 136], [246, 188, 148], [220, 161, 129], [183, 123, 97], [204, 160, 125], [224, 179, 138], [239, 189, 140], [252, 198, 138], [248, 189, 121], [249, 185, 113], [255, 198, 127], [247, 183, 119], [240, 180, 126], [246, 194, 154], [88, 48, 22], [232, 204, 190], [255, 252, 246], [255, 255, 255], [238, 247, 252], [242, 255, 255]], [[255, 251, 255], [255, 251, 255], [255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [249, 249, 249], [252, 250, 251], [250, 248, 249], [255, 254, 255], [195, 191, 192], [140, 73, 21], [255, 191, 136], [255, 193, 129], [255, 196, 124], [244, 186, 112], [244, 188, 114], [249, 194, 129], [251, 196, 142], [248, 190, 152], [195, 135, 107], [204, 144, 120], [130, 68, 45], [102, 39, 8], [233, 171, 130], [252, 190, 141], [255, 195, 140], [248, 193, 129], [252, 196, 135], [233, 177, 116], [255, 201, 142], [255, 203, 144], [226, 171, 107], [252, 197, 130], [248, 194, 122], [243, 189, 115], [255, 203, 129], [231, 177, 105], [245, 190, 126], [243, 186, 133], [243, 184, 144], [206, 145, 114], [182, 120, 95], [64, 13, 0], [174, 122, 85], [248, 194, 148], [247, 188, 132], [253, 189, 125], [245, 178, 108], [255, 197, 128], [255, 188, 125], [254, 191, 140], [251, 196, 157], [134, 90, 65], [215, 183, 170], [255, 243, 239], [250, 241, 244], [254, 255, 255], [248, 252, 255]], [[255, 251, 255], [255, 252, 255], [255, 252, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [247, 249, 248], [255, 255, 255], [255, 254, 255], [255, 254, 255], [253, 247, 249], [255, 252, 255], [175, 166, 169], [179, 114, 58], [255, 202, 141], [248, 187, 120], [249, 189, 116], [249, 190, 114], [250, 192, 118], [250, 190, 127], [246, 184, 133], [235, 172, 137], [192, 127, 105], [255, 217, 201], [182, 116, 102], [71, 6, 0], [192, 128, 100], [255, 215, 179], [231, 168, 125], [246, 190, 129], [251, 195, 136], [255, 209, 152], [231, 175, 118], [239, 183, 126], [253, 197, 138], [251, 195, 134], [239, 184, 119], [252, 198, 128], [248, 194, 124], [255, 204, 137], [254, 198, 137], [255, 197, 147], [200, 141, 101], [215, 154, 123], [255, 218, 193], [73, 16, 0], [116, 57, 25], [235, 173, 132], [255, 197, 144], [255, 193, 133], [255, 191, 126], [255, 188, 123], [250, 184, 124], [247, 185, 136], [255, 202, 167], [171, 125, 102], [183, 147, 135], [255, 247, 246], [255, 250, 253], [255, 250, 255], [255, 253, 255]], [[255, 252, 255], [255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [244, 246, 245], [255, 255, 255], [255, 254, 255], [251, 245, 247], [255, 251, 255], [255, 250, 254], [169, 154, 159], [159, 102, 49], [238, 182, 125], [241, 186, 121], [253, 196, 125], [255, 197, 123], [249, 189, 117], [253, 192, 129], [255, 196, 145], [216, 150, 116], [127, 59, 38], [143, 75, 64], [103, 35, 26], [73, 7, 0], [186, 124, 103], [248, 187, 159], [248, 189, 155], [248, 193, 136], [234, 179, 123], [246, 191, 137], [250, 194, 145], [255, 205, 156], [240, 184, 135], [245, 189, 138], [254, 199, 143], [234, 180, 120], [255, 204, 142], [241, 187, 127], [214, 159, 103], [255, 215, 168], [183, 125, 87], [91, 32, 2], [151, 91, 65], [106, 42, 17], [80, 15, 0], [223, 158, 120], [255, 194, 146], [247, 182, 124], [255, 194, 133], [247, 183, 122], [252, 191, 136], [237, 180, 135], [255, 205, 173], [158, 113, 92], [153, 116, 107], [255, 247, 244], [255, 246, 248], [255, 240, 245], [255, 251, 255]], [[255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [248, 252, 251], [254, 255, 255], [235, 233, 234], [255, 253, 255], [255, 251, 255], [255, 243, 248], [255, 249, 255], [117, 98, 104], [192, 147, 105], [243, 197, 148], [242, 195, 139], [248, 199, 133], [251, 198, 128], [239, 182, 113], [243, 181, 120], [254, 188, 138], [238, 170, 135], [95, 27, 6], [77, 9, 0], [129, 65, 55], [70, 8, 0], [190, 133, 114], [237, 183, 157], [242, 191, 160], [255, 206, 155], [243, 193, 144], [243, 192, 147], [221, 169, 129], [153, 101, 64], [135, 83, 46], [147, 95, 56], [227, 176, 133], [248, 197, 150], [243, 193, 144], [244, 194, 145], [248, 197, 150], [240, 189, 146], [198, 145, 111], [71, 18, 0], [62, 8, 0], [107, 40, 14], [93, 26, 0], [226, 161, 123], [255, 191, 145], [243, 182, 128], [253, 194, 136], [243, 187, 130], [249, 197, 147], [248, 199, 158], [231, 187, 158], [216, 178, 159], [109, 76, 67], [255, 236, 233], [255, 243, 245], [255, 249, 253], [255, 250, 253]], [[255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [246, 246, 246], [246, 242, 243], [255, 252, 255], [255, 250, 255], [255, 236, 242], [185, 164, 171], [110, 87, 95], [255, 230, 202], [255, 222, 189], [226, 189, 145], [221, 180, 126], [245, 199, 137], [246, 195, 130], [253, 194, 134], [254, 190, 142], [247, 181, 146], [162, 96, 70], [97, 31, 15], [106, 47, 33], [78, 25, 7], [202, 156, 132], [230, 189, 157], [231, 193, 157], [213, 172, 126], [251, 209, 167], [255, 215, 177], [230, 187, 155], [61, 17, 0], [86, 41, 18], [70, 25, 2], [207, 163, 136], [255, 216, 184], [247, 204, 169], [220, 178, 138], [219, 177, 139], [241, 198, 163], [209, 166, 134], [87, 43, 16], [106, 62, 37], [85, 15, 0], [142, 75, 46], [241, 177, 139], [254, 193, 148], [253, 197, 146], [241, 192, 136], [237, 191, 139], [231, 191, 142], [233, 195, 158], [225, 191, 164], [255, 237, 219], [147, 120, 111], [156, 132, 128], [255, 249, 248], [255, 237, 238], [255, 250, 250]], [[255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [237, 237, 237], [255, 253, 255], [255, 251, 255], [249, 232, 238], [199, 178, 185], [122, 96, 105], [214, 187, 196], [255, 241, 231], [242, 223, 206], [227, 202, 172], [217, 185, 144], [241, 201, 149], [241, 195, 136], [249, 194, 137], [251, 191, 141], [246, 183, 142], [217, 153, 125], [117, 59, 35], [74, 22, 0], [152, 108, 83], [228, 192, 160], [237, 208, 168], [247, 223, 179], [255, 245, 207], [243, 209, 174], [247, 212, 184], [255, 218, 199], [238, 200, 187], [66, 28, 19], [188, 150, 141], [255, 234, 223], [249, 212, 194], [234, 198, 174], [255, 248, 220], [255, 228, 198], [243, 208, 180], [214, 178, 152], [154, 118, 94], [76, 40, 18], [126, 59, 32], [213, 148, 116], [251, 191, 154], [238, 184, 138], [243, 196, 144], [221, 181, 129], [230, 197, 146], [221, 192, 148], [227, 203, 167], [247, 225, 201], [253, 232, 215], [241, 221, 212], [108, 89, 83], [183, 164, 160], [249, 230, 226], [255, 250, 246]], [[255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [249, 255, 255], [249, 255, 255], [251, 255, 255], [238, 242, 241], [255, 255, 255], [242, 236, 238], [255, 251, 255], [246, 227, 233], [121, 95, 104], [120, 90, 100], [255, 231, 242], [247, 240, 247], [246, 236, 234], [255, 243, 227], [242, 219, 187], [247, 214, 169], [231, 190, 136], [234, 183, 128], [234, 178, 127], [255, 202, 158], [235, 176, 142], [228, 175, 144], [226, 181, 152], [232, 196, 162], [240, 215, 174], [255, 252, 204], [249, 236, 183], [255, 241, 208], [206, 179, 150], [71, 43, 22], [186, 156, 145], [163, 132, 129], [63, 31, 34], [144, 112, 115], [195, 163, 164], [64, 33, 28], [178, 148, 137], [255, 237, 219], [247, 219, 197], [255, 230, 208], [250, 222, 200], [232, 204, 182], [195, 167, 146], [218, 151, 122], [255, 197, 162], [255, 198, 158], [231, 183, 137], [240, 200, 149], [221, 189, 138], [241, 218, 168], [237, 218, 176], [255, 243, 210], [252, 240, 216], [255, 250, 234], [251, 238, 229], [132, 117, 110], [82, 67, 60], [255, 251, 244], [255, 240, 232]], [[255, 255, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [249, 255, 255], [249, 255, 255], [251, 255, 255], [252, 255, 255], [251, 251, 251], [255, 253, 255], [222, 210, 214], [231, 210, 217], [168, 142, 151], [34, 4, 14], [224, 192, 203], [250, 249, 255], [242, 240, 245], [255, 250, 241], [247, 230, 204], [255, 241, 199], [255, 228, 177], [255, 222, 168], [255, 210, 158], [247, 190, 145], [255, 217, 180], [249, 199, 166], [255, 221, 187], [255, 232, 194], [255, 241, 196], [251, 240, 186], [255, 247, 190], [255, 247, 216], [247, 223, 197], [223, 197, 182], [164, 137, 130], [195, 166, 168], [229, 199, 207], [211, 181, 191], [160, 131, 136], [206, 177, 179], [255, 234, 227], [255, 238, 225], [255, 242, 225], [253, 228, 208], [255, 244, 224], [255, 233, 213], [246, 221, 201], [255, 204, 172], [255, 199, 164], [255, 200, 159], [247, 200, 154], [255, 225, 173], [248, 220, 170], [255, 242, 194], [255, 244, 202], [254, 245, 212], [244, 236, 213], [249, 242, 226], [224, 214, 204], [35, 25, 16], [129, 114, 107], [211, 197, 188], [233, 216, 206]], [[251, 255, 253], [251, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 254, 251], [244, 234, 232], [255, 251, 248], [147, 128, 122], [30, 7, 1], [190, 163, 154], [235, 207, 196], [255, 230, 220], [238, 251, 231], [239, 252, 234], [225, 239, 224], [223, 239, 228], [233, 250, 242], [230, 249, 247], [228, 246, 248], [232, 247, 252], [234, 245, 251], [236, 243, 249], [241, 242, 247], [246, 240, 244], [250, 238, 240], [255, 237, 237], [255, 235, 233], [255, 234, 233], [245, 246, 241], [246, 247, 242], [247, 248, 243], [249, 248, 244], [250, 249, 244], [251, 250, 245], [252, 251, 246], [253, 253, 245], [243, 243, 235], [243, 243, 233], [243, 243, 233], [242, 244, 233], [241, 245, 231], [241, 245, 231], [241, 245, 231], [239, 245, 231], [234, 240, 230], [251, 255, 247], [243, 249, 239], [231, 234, 225], [243, 244, 236], [246, 246, 236], [243, 240, 231], [254, 247, 239], [249, 236, 228], [245, 228, 221], [255, 236, 232], [255, 239, 237], [239, 205, 203], [214, 176, 175], [57, 15, 16], [113, 69, 70]], [[252, 255, 253], [252, 255, 253], [252, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [255, 253, 253], [249, 239, 238], [255, 241, 238], [249, 231, 227], [56, 36, 29], [208, 183, 176], [255, 248, 239], [255, 242, 234], [229, 233, 216], [251, 255, 242], [239, 247, 232], [238, 248, 237], [247, 255, 250], [225, 237, 233], [222, 234, 234], [247, 255, 255], [236, 244, 247], [238, 243, 246], [242, 242, 244], [245, 241, 242], [249, 239, 238], [253, 238, 235], [255, 237, 233], [255, 237, 231], [239, 240, 235], [239, 240, 235], [241, 240, 235], [242, 241, 236], [242, 241, 236], [243, 242, 237], [243, 243, 235], [244, 244, 236], [243, 243, 235], [243, 243, 233], [242, 244, 233], [242, 244, 233], [241, 244, 233], [241, 245, 231], [239, 245, 231], [239, 245, 231], [239, 242, 233], [228, 231, 222], [237, 240, 231], [245, 246, 238], [251, 253, 242], [237, 237, 227], [240, 236, 227], [237, 230, 222], [242, 231, 225], [255, 251, 244], [255, 240, 235], [255, 237, 234], [255, 246, 244], [213, 177, 177], [68, 29, 30], [246, 206, 207]], [[254, 255, 253], [254, 255, 253], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [254, 250, 249], [255, 253, 251], [251, 241, 239], [255, 247, 244], [241, 223, 219], [72, 51, 46], [128, 105, 99], [154, 131, 123], [200, 194, 180], [231, 225, 213], [255, 255, 244], [249, 246, 237], [238, 238, 230], [247, 248, 242], [238, 240, 235], [230, 232, 229], [241, 243, 240], [242, 242, 240], [243, 242, 238], [245, 242, 237], [246, 241, 235], [248, 241, 233], [249, 240, 231], [250, 240, 231], [242, 241, 236], [242, 241, 236], [242, 241, 236], [242, 241, 236], [242, 242, 234], [242, 242, 234], [242, 242, 234], [242, 242, 234], [243, 243, 235], [242, 244, 233], [242, 244, 233], [241, 244, 233], [241, 244, 233], [239, 245, 233], [239, 245, 233], [239, 245, 233], [234, 234, 224], [252, 252, 242], [250, 250, 240], [221, 221, 211], [252, 249, 240], [250, 247, 238], [253, 249, 240], [251, 244, 236], [249, 238, 232], [255, 247, 242], [199, 180, 176], [156, 132, 130], [139, 111, 110], [76, 44, 45], [236, 202, 203], [255, 232, 234]], [[255, 255, 253], [255, 255, 253], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [248, 246, 247], [255, 251, 252], [250, 241, 242], [255, 252, 250], [255, 250, 247], [255, 247, 242], [114, 96, 92], [53, 36, 29], [229, 210, 203], [255, 251, 243], [230, 213, 205], [247, 232, 225], [251, 238, 230], [232, 221, 215], [255, 253, 246], [238, 231, 223], [246, 242, 233], [245, 242, 233], [245, 242, 233], [243, 243, 233], [243, 243, 233], [243, 243, 231], [243, 243, 231], [243, 243, 231], [249, 246, 239], [248, 245, 238], [248, 245, 238], [246, 246, 238], [245, 245, 237], [245, 245, 237], [244, 244, 236], [244, 244, 236], [242, 243, 235], [242, 243, 235], [241, 244, 235], [241, 244, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [255, 248, 238], [241, 237, 226], [248, 244, 233], [242, 238, 229], [255, 255, 246], [230, 226, 217], [240, 233, 225], [252, 245, 237], [242, 233, 226], [255, 245, 242], [245, 230, 227], [57, 37, 36], [55, 31, 31], [255, 230, 231], [255, 243, 245], [255, 246, 248]], [[255, 254, 251], [255, 254, 251], [255, 255, 253], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [251, 249, 252], [255, 254, 255], [240, 234, 234], [250, 242, 240], [255, 252, 250], [232, 222, 220], [38, 27, 23], [190, 165, 161], [240, 215, 211], [255, 242, 238], [255, 249, 243], [255, 249, 243], [250, 230, 221], [236, 218, 208], [255, 248, 237], [252, 240, 228], [249, 241, 228], [246, 242, 230], [243, 244, 230], [241, 245, 231], [239, 245, 231], [238, 246, 231], [238, 246, 233], [248, 244, 235], [247, 243, 234], [246, 243, 234], [245, 242, 233], [244, 241, 232], [242, 242, 232], [241, 241, 231], [240, 242, 231], [242, 244, 233], [241, 244, 235], [241, 244, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [255, 251, 242], [238, 228, 219], [250, 240, 231], [245, 236, 227], [247, 238, 229], [249, 242, 234], [255, 254, 246], [243, 236, 230], [255, 253, 248], [224, 215, 210], [180, 169, 167], [75, 59, 59], [189, 171, 171], [255, 247, 249], [255, 247, 249], [255, 245, 248]], [[255, 253, 251], [255, 254, 251], [255, 253, 253], [255, 254, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [251, 252, 255], [250, 251, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [254, 250, 249], [243, 239, 238], [245, 240, 237], [139, 113, 116], [90, 64, 65], [150, 124, 123], [166, 141, 137], [187, 162, 157], [255, 241, 234], [255, 244, 233], [251, 232, 218], [255, 239, 224], [252, 240, 224], [248, 242, 226], [243, 244, 228], [241, 245, 231], [238, 246, 233], [235, 247, 235], [234, 248, 235], [249, 242, 232], [247, 243, 232], [246, 242, 231], [246, 242, 231], [244, 242, 230], [242, 242, 230], [241, 241, 231], [240, 242, 231], [241, 244, 233], [241, 244, 235], [241, 244, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 244, 237], [239, 244, 237], [244, 227, 219], [255, 251, 243], [253, 239, 230], [248, 235, 227], [255, 253, 244], [235, 226, 219], [182, 173, 166], [184, 177, 171], [139, 131, 128], [89, 81, 78], [119, 111, 109], [240, 230, 229], [255, 244, 246], [255, 251, 253], [253, 238, 243], [255, 250, 255]], [[255, 252, 250], [255, 253, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [229, 233, 236], [254, 255, 255], [250, 251, 253], [244, 244, 244], [255, 255, 255], [249, 249, 249], [255, 248, 254], [250, 229, 234], [202, 182, 184], [184, 163, 162], [113, 89, 85], [116, 91, 84], [245, 223, 212], [255, 245, 232], [255, 238, 224], [253, 240, 224], [249, 242, 226], [246, 242, 230], [242, 244, 233], [238, 246, 235], [235, 246, 238], [234, 247, 238], [251, 243, 232], [250, 244, 232], [250, 244, 232], [248, 244, 232], [247, 245, 233], [244, 244, 232], [244, 244, 232], [243, 245, 234], [241, 244, 233], [241, 244, 233], [241, 244, 235], [241, 244, 235], [241, 244, 235], [241, 244, 237], [241, 244, 237], [241, 244, 237], [249, 229, 222], [255, 243, 236], [246, 227, 220], [255, 251, 243], [230, 217, 209], [147, 136, 130], [92, 83, 78], [154, 147, 141], [184, 179, 175], [219, 214, 211], [255, 254, 253], [237, 231, 233], [255, 253, 255], [255, 253, 255], [255, 252, 255], [255, 249, 254]], [[255, 252, 250], [255, 252, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [248, 251, 255], [251, 255, 255], [251, 255, 255], [247, 251, 254], [251, 255, 255], [254, 255, 255], [242, 243, 245], [250, 252, 251], [255, 252, 255], [241, 231, 239], [255, 251, 255], [255, 244, 247], [255, 251, 248], [171, 152, 146], [47, 28, 21], [201, 181, 172], [255, 238, 226], [255, 238, 228], [252, 239, 230], [248, 241, 233], [245, 242, 235], [242, 243, 238], [239, 244, 240], [238, 244, 242], [250, 240, 230], [249, 241, 230], [249, 241, 230], [248, 242, 230], [246, 242, 230], [245, 243, 231], [243, 243, 231], [242, 244, 231], [242, 244, 233], [242, 244, 233], [242, 244, 233], [242, 243, 235], [242, 243, 235], [243, 243, 235], [243, 243, 235], [243, 243, 235], [255, 244, 239], [255, 239, 233], [255, 244, 237], [206, 187, 181], [47, 32, 25], [143, 132, 126], [255, 253, 248], [255, 253, 249], [253, 249, 246], [255, 255, 253], [255, 254, 255], [250, 248, 249], [255, 254, 255], [247, 245, 250], [250, 248, 253], [255, 253, 255]], [[255, 252, 250], [255, 252, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [250, 255, 255], [248, 253, 255], [249, 253, 255], [250, 254, 255], [253, 254, 255], [252, 253, 255], [251, 253, 252], [248, 252, 255], [247, 250, 255], [253, 254, 255], [255, 253, 255], [255, 253, 253], [255, 252, 250], [205, 191, 188], [104, 89, 84], [100, 83, 76], [215, 198, 191], [242, 227, 222], [255, 246, 245], [255, 252, 251], [231, 222, 223], [255, 251, 255], [244, 238, 242], [253, 243, 233], [242, 232, 222], [244, 236, 225], [251, 245, 233], [243, 239, 227], [238, 236, 224], [244, 244, 232], [250, 250, 238], [247, 247, 235], [219, 219, 209], [255, 255, 246], [244, 241, 232], [247, 243, 234], [246, 242, 233], [249, 242, 234], [255, 252, 246], [255, 237, 232], [224, 199, 194], [118, 97, 92], [77, 58, 52], [219, 204, 199], [255, 252, 248], [252, 244, 241], [255, 254, 251], [255, 255, 253], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255]], [[255, 252, 250], [255, 253, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [251, 254, 255], [251, 254, 255], [252, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [244, 255, 255], [242, 255, 255], [248, 255, 255], [251, 255, 255], [242, 243, 245], [252, 250, 251], [255, 253, 253], [255, 248, 247], [214, 200, 199], [79, 63, 63], [118, 100, 100], [184, 165, 167], [210, 189, 194], [255, 249, 255], [244, 225, 231], [255, 239, 245], [255, 246, 238], [255, 243, 235], [254, 244, 235], [255, 247, 238], [255, 251, 241], [254, 250, 239], [244, 242, 230], [235, 233, 221], [250, 248, 236], [255, 253, 241], [255, 255, 244], [233, 226, 216], [255, 253, 244], [255, 252, 244], [219, 204, 197], [192, 177, 170], [122, 97, 93], [81, 57, 55], [188, 167, 164], [255, 250, 246], [255, 247, 244], [237, 226, 224], [255, 253, 250], [255, 252, 249], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255]], [[255, 253, 251], [255, 254, 251], [255, 253, 253], [255, 254, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [252, 252, 254], [250, 250, 252], [251, 249, 250], [255, 251, 250], [255, 254, 253], [255, 254, 251], [239, 255, 255], [239, 255, 255], [245, 255, 255], [248, 255, 255], [251, 255, 255], [252, 253, 255], [255, 254, 255], [255, 250, 252], [254, 242, 246], [245, 229, 232], [47, 26, 31], [127, 102, 106], [109, 80, 85], [103, 72, 78], [154, 121, 128], [205, 172, 179], [206, 191, 186], [227, 214, 208], [236, 225, 219], [234, 225, 216], [235, 228, 218], [234, 230, 219], [235, 231, 220], [242, 240, 228], [232, 228, 217], [215, 209, 197], [198, 190, 179], [179, 166, 157], [170, 153, 145], [97, 78, 71], [87, 64, 58], [166, 143, 137], [102, 81, 80], [218, 197, 196], [255, 249, 248], [254, 238, 238], [255, 251, 250], [255, 252, 251], [255, 254, 251], [250, 246, 245], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255]], [[255, 254, 251], [255, 254, 251], [255, 255, 253], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [251, 251, 253], [255, 254, 255], [255, 254, 255], [255, 253, 253], [255, 253, 251], [255, 252, 250], [255, 248, 246], [255, 246, 242], [249, 255, 253], [249, 255, 253], [241, 250, 247], [233, 239, 239], [240, 244, 245], [254, 255, 255], [255, 254, 255], [255, 248, 255], [255, 251, 255], [255, 248, 254], [132, 109, 115], [223, 192, 197], [255, 244, 248], [227, 187, 188], [208, 164, 165], [165, 119, 121], [117, 102, 99], [112, 98, 95], [97, 83, 80], [94, 85, 78], [111, 104, 96], [114, 107, 99], [101, 97, 86], [102, 95, 85], [102, 96, 84], [114, 104, 94], [123, 109, 98], [146, 128, 118], [187, 165, 154], [218, 191, 182], [249, 219, 211], [255, 233, 226], [143, 124, 126], [199, 180, 182], [255, 241, 242], [255, 251, 251], [255, 247, 246], [251, 242, 243], [255, 254, 253], [252, 250, 251], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 254, 255], [255, 254, 255]], [[255, 255, 253], [255, 255, 253], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 249, 250], [254, 244, 242], [255, 241, 238], [255, 244, 239], [255, 250, 246], [255, 251, 244], [249, 242, 232], [255, 254, 244], [255, 254, 248], [255, 254, 251], [255, 255, 253], [255, 254, 255], [255, 251, 255], [249, 242, 249], [255, 251, 255], [249, 230, 236], [216, 191, 194], [73, 39, 40], [244, 203, 201], [255, 210, 204], [252, 199, 191], [255, 232, 222], [249, 233, 233], [237, 223, 222], [225, 211, 210], [223, 212, 210], [224, 215, 210], [216, 209, 201], [214, 207, 197], [223, 214, 205], [218, 208, 198], [249, 235, 224], [246, 228, 216], [255, 244, 232], [255, 247, 237], [255, 245, 235], [255, 242, 233], [255, 221, 213], [44, 29, 32], [239, 224, 227], [255, 251, 253], [236, 226, 227], [255, 252, 255], [255, 254, 255], [246, 244, 245], [255, 255, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255]], [[254, 255, 253], [254, 255, 253], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 253], [255, 253, 251], [255, 252, 250], [255, 251, 248], [255, 250, 246], [255, 244, 239], [255, 240, 234], [255, 239, 231], [255, 248, 233], [255, 249, 235], [255, 247, 236], [255, 251, 244], [255, 251, 250], [255, 251, 252], [255, 248, 253], [255, 251, 255], [250, 237, 244], [255, 249, 253], [255, 247, 248], [72, 37, 33], [93, 49, 40], [89, 37, 23], [162, 105, 86], [255, 239, 220], [255, 245, 248], [255, 240, 243], [255, 251, 251], [255, 252, 250], [252, 242, 240], [248, 239, 234], [255, 248, 241], [255, 249, 240], [255, 247, 235], [255, 246, 234], [243, 221, 208], [255, 232, 221], [252, 216, 204], [255, 238, 228], [255, 218, 207], [127, 80, 70], [155, 144, 150], [255, 251, 255], [255, 252, 255], [255, 253, 255], [255, 253, 255], [251, 249, 252], [249, 249, 251], [255, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255]], [[252, 255, 253], [252, 255, 253], [252, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [247, 241, 241], [255, 247, 246], [255, 251, 248], [255, 244, 240], [245, 225, 218], [223, 198, 191], [204, 177, 168], [194, 165, 157], [202, 161, 141], [212, 174, 155], [215, 182, 167], [234, 205, 197], [255, 239, 234], [255, 246, 247], [254, 242, 246], [255, 250, 255], [255, 240, 245], [255, 249, 250], [203, 175, 172], [126, 89, 80], [192, 146, 130], [136, 81, 60], [65, 5, 0], [226, 163, 132], [246, 229, 235], [242, 226, 229], [255, 244, 247], [255, 244, 244], [235, 225, 223], [252, 243, 238], [255, 253, 246], [232, 222, 213], [255, 252, 241], [224, 204, 193], [64, 37, 26], [132, 100, 87], [255, 230, 218], [231, 184, 174], [117, 67, 56], [69, 17, 6], [213, 206, 213], [255, 252, 255], [247, 240, 247], [255, 253, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [243, 244, 246], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255], [255, 253, 255], [255, 252, 255]], [[251, 255, 253], [251, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 254, 251], [229, 219, 217], [170, 155, 152], [113, 94, 88], [84, 61, 55], [91, 64, 55], [121, 93, 82], [147, 117, 107], [148, 98, 75], [156, 108, 88], [121, 78, 62], [86, 49, 40], [112, 84, 80], [171, 149, 151], [221, 205, 208], [255, 243, 248], [255, 244, 247], [255, 249, 250], [192, 164, 160], [143, 107, 93], [254, 207, 187], [229, 175, 147], [224, 164, 130], [112, 48, 12], [120, 103, 109], [186, 169, 175], [255, 246, 249], [232, 220, 222], [150, 140, 139], [173, 164, 159], [247, 238, 231], [255, 248, 240], [253, 236, 226], [163, 143, 132], [104, 76, 64], [139, 105, 93], [65, 23, 11], [99, 51, 39], [214, 162, 151], [174, 120, 108], [148, 141, 148], [255, 253, 255], [255, 252, 255], [250, 248, 253], [255, 254, 255], [255, 254, 255], [251, 252, 254], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255], [255, 252, 253], [255, 252, 253]], [[254, 255, 255], [252, 255, 255], [252, 255, 255], [252, 255, 255], [250, 250, 250], [255, 252, 246], [255, 245, 230], [232, 205, 178], [142, 103, 64], [107, 55, 5], [145, 85, 25], [227, 159, 94], [254, 184, 114], [251, 181, 109], [255, 192, 122], [255, 200, 129], [237, 205, 118], [245, 210, 126], [239, 203, 125], [239, 202, 131], [200, 160, 98], [153, 112, 58], [100, 56, 9], [121, 76, 35], [245, 200, 161], [255, 232, 193], [178, 133, 92], [184, 139, 97], [231, 187, 140], [239, 198, 146], [249, 208, 152], [230, 190, 131], [214, 154, 118], [92, 29, 0], [157, 93, 58], [147, 78, 45], [138, 69, 40], [124, 54, 29], [255, 201, 182], [255, 212, 195], [141, 84, 67], [128, 75, 57], [239, 187, 163], [246, 193, 159], [207, 153, 106], [252, 193, 135], [255, 198, 129], [218, 154, 80], [136, 79, 85], [255, 230, 235], [255, 242, 250], [255, 246, 253], [255, 244, 251], [255, 253, 255], [254, 255, 255], [249, 253, 252], [252, 255, 251], [255, 255, 250], [255, 255, 250], [255, 254, 251], [255, 253, 255], [255, 252, 255], [255, 252, 255], [255, 252, 255]], [[254, 252, 255], [246, 245, 250], [255, 254, 255], [241, 241, 243], [255, 254, 252], [242, 235, 227], [137, 121, 106], [89, 63, 36], [196, 158, 121], [215, 165, 116], [248, 189, 133], [255, 200, 138], [255, 191, 125], [255, 196, 128], [255, 203, 134], [253, 190, 123], [244, 201, 123], [240, 196, 121], [229, 185, 114], [248, 200, 138], [251, 202, 146], [242, 192, 143], [223, 170, 128], [217, 163, 125], [68, 14, 0], [115, 61, 25], [82, 28, 0], [206, 153, 111], [255, 215, 168], [235, 185, 134], [237, 188, 132], [236, 187, 129], [195, 137, 100], [242, 182, 145], [159, 97, 60], [228, 164, 128], [249, 183, 149], [149, 84, 54], [175, 111, 86], [140, 79, 58], [183, 128, 107], [222, 170, 148], [166, 116, 89], [217, 167, 130], [250, 198, 150], [246, 190, 131], [255, 194, 127], [235, 173, 100], [98, 40, 39], [105, 53, 55], [237, 196, 200], [255, 246, 251], [255, 250, 255], [255, 252, 255], [240, 241, 245], [252, 255, 255], [252, 255, 253], [255, 255, 250], [255, 255, 250], [255, 254, 251], [255, 254, 255], [255, 253, 255], [255, 253, 255], [255, 252, 255]], [[255, 246, 245], [255, 252, 253], [255, 253, 253], [253, 249, 248], [191, 186, 183], [24, 15, 8], [175, 158, 142], [244, 218, 193], [236, 200, 164], [242, 198, 153], [247, 196, 141], [249, 193, 134], [247, 188, 128], [243, 187, 126], [242, 188, 126], [239, 187, 127], [238, 180, 116], [244, 186, 123], [239, 180, 122], [246, 186, 134], [253, 190, 146], [254, 190, 152], [255, 206, 171], [228, 163, 131], [139, 74, 42], [194, 129, 97], [128, 64, 28], [237, 174, 133], [255, 196, 149], [236, 176, 124], [250, 191, 135], [226, 167, 109], [84, 29, 0], [228, 173, 132], [255, 197, 157], [250, 191, 149], [236, 174, 135], [182, 122, 85], [95, 36, 2], [207, 152, 121], [223, 172, 143], [255, 220, 190], [111, 65, 31], [198, 150, 110], [255, 215, 166], [238, 186, 126], [244, 187, 120], [245, 187, 114], [128, 68, 58], [189, 134, 127], [53, 9, 6], [230, 198, 199], [255, 247, 249], [255, 253, 255], [255, 255, 255], [253, 254, 255], [254, 255, 255], [255, 255, 251], [255, 255, 251], [255, 254, 251], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255]], [[255, 247, 243], [255, 251, 248], [232, 218, 215], [127, 116, 114], [80, 69, 65], [234, 221, 212], [253, 236, 220], [254, 228, 203], [232, 198, 163], [229, 187, 145], [224, 178, 128], [234, 184, 131], [245, 196, 140], [239, 192, 136], [234, 189, 134], [242, 199, 146], [251, 184, 131], [255, 196, 144], [255, 194, 147], [255, 187, 145], [252, 180, 142], [241, 168, 136], [255, 182, 153], [165, 91, 64], [191, 117, 88], [250, 179, 147], [150, 80, 44], [238, 169, 127], [245, 177, 128], [255, 189, 136], [255, 216, 158], [225, 161, 99], [134, 81, 39], [234, 181, 137], [252, 198, 152], [243, 189, 142], [255, 210, 160], [247, 195, 147], [216, 165, 118], [254, 205, 162], [242, 197, 156], [235, 190, 151], [125, 80, 39], [203, 159, 114], [244, 197, 145], [242, 192, 133], [240, 186, 122], [244, 189, 122], [107, 45, 24], [255, 211, 194], [181, 133, 121], [141, 106, 100], [255, 249, 248], [243, 233, 234], [241, 239, 242], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 255, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255]], [[255, 248, 241], [240, 217, 211], [58, 39, 33], [180, 162, 158], [240, 225, 218], [255, 245, 235], [255, 235, 218], [228, 202, 177], [217, 182, 150], [242, 203, 162], [249, 205, 158], [240, 194, 144], [238, 195, 142], [239, 198, 146], [232, 195, 143], [229, 193, 143], [255, 188, 139], [255, 188, 142], [252, 183, 141], [253, 184, 143], [255, 191, 155], [255, 185, 153], [248, 177, 147], [117, 46, 16], [235, 164, 134], [247, 179, 144], [127, 59, 20], [236, 169, 124], [250, 184, 132], [237, 172, 114], [255, 192, 130], [217, 153, 89], [133, 82, 39], [233, 182, 135], [243, 193, 142], [247, 198, 142], [238, 189, 130], [211, 165, 105], [255, 217, 158], [233, 190, 135], [247, 204, 153], [214, 170, 123], [145, 101, 54], [211, 165, 116], [240, 193, 141], [246, 197, 139], [249, 199, 138], [237, 186, 121], [131, 65, 31], [212, 151, 122], [255, 217, 194], [61, 23, 10], [255, 240, 235], [255, 251, 251], [255, 253, 255], [236, 235, 240], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 253], [255, 255, 255], [254, 255, 255], [251, 255, 255], [251, 255, 255]], [[236, 207, 199], [83, 56, 49], [192, 169, 163], [255, 238, 233], [254, 235, 228], [255, 245, 234], [255, 245, 227], [255, 228, 201], [255, 224, 189], [246, 204, 162], [236, 192, 143], [239, 193, 141], [235, 192, 139], [229, 188, 136], [226, 189, 137], [230, 194, 144], [243, 182, 128], [245, 183, 132], [245, 183, 134], [246, 185, 140], [246, 184, 143], [251, 189, 152], [228, 168, 132], [112, 52, 16], [252, 192, 156], [240, 181, 141], [117, 58, 14], [222, 164, 114], [255, 205, 150], [248, 192, 131], [255, 211, 147], [249, 194, 129], [136, 81, 40], [207, 155, 108], [255, 205, 152], [237, 191, 131], [230, 185, 118], [255, 220, 151], [226, 186, 116], [240, 199, 133], [245, 200, 141], [232, 187, 132], [130, 82, 33], [207, 157, 108], [255, 205, 156], [242, 191, 138], [253, 202, 147], [228, 176, 119], [137, 66, 22], [229, 166, 125], [233, 180, 148], [98, 57, 37], [214, 187, 178], [255, 244, 244], [255, 253, 255], [254, 251, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [254, 255, 253], [251, 255, 253], [249, 255, 255], [248, 255, 255]], [[66, 36, 28], [166, 137, 131], [246, 221, 216], [255, 249, 243], [240, 220, 213], [242, 220, 207], [255, 246, 226], [255, 248, 220], [255, 248, 211], [239, 195, 150], [228, 178, 127], [247, 196, 141], [246, 197, 139], [238, 191, 135], [241, 198, 143], [246, 205, 149], [244, 195, 129], [249, 199, 136], [253, 203, 144], [251, 202, 146], [233, 183, 132], [244, 196, 148], [206, 159, 113], [126, 79, 35], [236, 189, 145], [240, 193, 147], [136, 90, 41], [203, 157, 105], [253, 206, 150], [236, 190, 130], [232, 186, 124], [209, 164, 99], [128, 69, 29], [227, 170, 125], [248, 195, 143], [240, 192, 130], [230, 188, 116], [244, 202, 126], [217, 175, 99], [243, 201, 129], [246, 198, 134], [233, 182, 125], [118, 65, 15], [223, 167, 120], [253, 196, 151], [243, 187, 140], [248, 195, 145], [224, 171, 121], [138, 67, 13], [231, 165, 117], [255, 204, 167], [115, 73, 48], [232, 202, 191], [255, 250, 248], [245, 236, 241], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [254, 255, 255], [252, 255, 253], [251, 255, 253], [248, 255, 253], [247, 255, 253]], [[223, 195, 191], [76, 51, 46], [197, 173, 169], [255, 249, 244], [255, 250, 243], [255, 242, 229], [255, 236, 218], [248, 213, 185], [235, 191, 152], [236, 186, 137], [255, 201, 145], [255, 199, 139], [238, 180, 117], [246, 191, 127], [254, 203, 140], [230, 180, 117], [245, 202, 124], [234, 192, 116], [231, 189, 115], [238, 198, 129], [231, 192, 127], [255, 216, 157], [193, 154, 99], [120, 80, 29], [238, 198, 149], [233, 193, 144], [135, 95, 46], [187, 145, 95], [247, 206, 154], [230, 187, 134], [237, 194, 139], [247, 202, 147], [138, 72, 37], [220, 157, 116], [253, 195, 145], [238, 188, 127], [245, 201, 130], [243, 201, 125], [233, 190, 112], [248, 201, 129], [254, 203, 138], [213, 154, 98], [134, 72, 25], [255, 194, 152], [238, 175, 134], [255, 194, 155], [247, 188, 146], [231, 174, 131], [137, 65, 6], [234, 169, 115], [255, 211, 170], [117, 73, 44], [216, 187, 173], [255, 250, 246], [255, 252, 255], [255, 253, 255], [255, 253, 255], [255, 254, 255], [254, 255, 255], [254, 255, 255], [252, 255, 253], [249, 255, 251], [248, 255, 251], [247, 255, 251]], [[255, 248, 246], [220, 199, 198], [45, 25, 24], [174, 156, 154], [255, 237, 231], [255, 245, 234], [247, 219, 198], [231, 194, 165], [237, 189, 149], [246, 190, 139], [252, 190, 131], [255, 190, 124], [255, 189, 121], [253, 189, 118], [252, 189, 120], [251, 191, 121], [238, 194, 107], [253, 208, 123], [238, 196, 114], [228, 187, 108], [240, 200, 128], [248, 207, 143], [202, 162, 103], [118, 77, 23], [233, 191, 141], [241, 198, 153], [147, 102, 60], [180, 133, 91], [250, 201, 160], [228, 176, 136], [246, 193, 151], [252, 199, 157], [180, 105, 76], [189, 119, 85], [255, 197, 153], [245, 193, 135], [248, 201, 133], [240, 196, 123], [232, 186, 111], [248, 198, 129], [250, 191, 131], [211, 145, 95], [149, 76, 35], [255, 201, 166], [248, 175, 143], [252, 183, 150], [255, 194, 159], [200, 137, 102], [143, 73, 13], [255, 204, 150], [244, 191, 149], [98, 57, 27], [224, 196, 182], [255, 250, 246], [255, 248, 253], [255, 253, 255], [255, 253, 255], [254, 254, 255], [254, 255, 255], [254, 255, 255], [252, 255, 253], [249, 255, 251], [248, 255, 251], [247, 255, 251]], [[255, 239, 240], [255, 250, 251], [241, 227, 227], [101, 87, 87], [117, 102, 99], [238, 220, 210], [255, 249, 230], [234, 197, 168], [240, 193, 151], [241, 184, 131], [243, 177, 117], [250, 181, 114], [255, 190, 120], [255, 195, 123], [255, 192, 121], [253, 185, 114], [245, 191, 103], [255, 204, 118], [238, 186, 103], [248, 196, 120], [246, 196, 125], [249, 199, 136], [226, 175, 118], [105, 53, 5], [221, 168, 124], [250, 194, 157], [212, 153, 121], [97, 33, 5], [233, 168, 140], [227, 160, 134], [198, 128, 103], [236, 164, 140], [193, 111, 90], [177, 101, 75], [237, 172, 134], [247, 191, 140], [241, 193, 131], [240, 195, 128], [248, 200, 134], [236, 182, 122], [255, 195, 145], [179, 104, 64], [201, 120, 90], [239, 158, 131], [251, 171, 148], [241, 167, 142], [255, 193, 166], [174, 109, 81], [185, 118, 63], [244, 184, 134], [240, 191, 151], [93, 56, 29], [253, 227, 214], [255, 251, 248], [238, 232, 236], [254, 253, 255], [254, 254, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [252, 255, 253], [251, 255, 253], [249, 255, 253], [248, 255, 253]], [[255, 251, 255], [250, 239, 243], [254, 245, 248], [255, 249, 252], [164, 154, 153], [36, 21, 14], [203, 180, 164], [238, 204, 177], [244, 199, 160], [255, 200, 151], [255, 198, 142], [255, 191, 129], [254, 183, 117], [250, 180, 111], [253, 184, 117], [255, 190, 122], [252, 189, 110], [255, 202, 125], [245, 183, 110], [255, 197, 127], [235, 173, 112], [239, 178, 123], [255, 201, 154], [154, 91, 50], [212, 146, 112], [252, 182, 156], [171, 96, 75], [100, 21, 6], [97, 13, 2], [104, 17, 8], [136, 46, 38], [130, 39, 34], [132, 46, 33], [120, 42, 22], [244, 178, 146], [255, 209, 165], [224, 178, 126], [229, 187, 129], [235, 188, 132], [247, 191, 142], [251, 183, 144], [145, 66, 36], [122, 35, 15], [171, 84, 67], [184, 101, 85], [185, 108, 90], [230, 162, 141], [128, 64, 39], [209, 147, 100], [255, 214, 170], [191, 145, 111], [122, 90, 67], [253, 233, 222], [252, 242, 240], [255, 254, 255], [253, 254, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [254, 255, 253], [252, 255, 253], [251, 255, 255], [249, 255, 255]], [[255, 253, 255], [255, 250, 254], [255, 253, 255], [248, 243, 247], [255, 253, 255], [226, 217, 212], [95, 77, 65], [81, 53, 32], [187, 148, 117], [216, 167, 126], [248, 192, 143], [255, 203, 148], [255, 199, 142], [254, 192, 135], [250, 189, 132], [251, 190, 135], [252, 184, 121], [252, 184, 123], [237, 171, 113], [255, 196, 142], [255, 196, 148], [255, 193, 151], [252, 186, 151], [167, 100, 71], [132, 60, 38], [209, 133, 119], [77, 0, 0], [118, 30, 26], [118, 24, 24], [130, 31, 34], [147, 46, 52], [126, 22, 29], [124, 39, 32], [84, 6, 0], [219, 153, 131], [246, 193, 159], [239, 196, 153], [255, 229, 183], [227, 182, 140], [253, 197, 162], [231, 161, 135], [86, 4, 0], [126, 37, 29], [115, 24, 19], [91, 6, 0], [128, 52, 39], [101, 34, 17], [128, 67, 46], [202, 146, 109], [204, 154, 121], [151, 111, 85], [202, 175, 158], [255, 247, 239], [255, 254, 251], [250, 251, 253], [243, 247, 250], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 255, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255]], [[254, 250, 251], [249, 247, 250], [244, 243, 248], [253, 252, 255], [235, 235, 237], [255, 254, 253], [255, 244, 238], [219, 200, 185], [123, 93, 69], [138, 99, 68], [163, 118, 79], [200, 151, 110], [234, 185, 142], [253, 204, 163], [253, 205, 165], [244, 199, 158], [252, 191, 147], [255, 200, 158], [255, 204, 164], [248, 190, 153], [222, 166, 133], [182, 125, 96], [142, 84, 62], [141, 80, 62], [101, 37, 25], [86, 15, 9], [114, 39, 36], [113, 31, 33], [130, 41, 47], [131, 36, 44], [126, 26, 36], [114, 12, 25], [128, 44, 44], [96, 21, 15], [149, 87, 72], [187, 140, 114], [200, 164, 130], [214, 180, 143], [194, 155, 122], [229, 177, 155], [173, 107, 93], [85, 4, 1], [137, 48, 50], [107, 18, 22], [125, 44, 41], [117, 44, 37], [89, 27, 12], [128, 73, 53], [114, 64, 39], [81, 36, 15], [59, 26, 9], [237, 215, 204], [255, 252, 248], [239, 238, 236], [251, 255, 255], [249, 255, 255], [251, 255, 255], [254, 255, 255], [255, 255, 253], [255, 255, 253], [255, 254, 255], [255, 254, 255], [255, 253, 255], [255, 254, 255]], [[255, 254, 253], [255, 254, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 254, 251], [255, 245, 236], [254, 235, 220], [217, 190, 169], [159, 127, 102], [115, 81, 53], [103, 69, 42], [117, 85, 60], [143, 113, 89], [160, 132, 110], [167, 125, 101], [159, 117, 93], [146, 105, 83], [110, 69, 49], [118, 80, 61], [147, 108, 91], [186, 146, 134], [255, 242, 233], [220, 173, 167], [66, 13, 9], [129, 69, 69], [124, 58, 60], [103, 28, 33], [103, 22, 29], [131, 45, 54], [143, 56, 65], [108, 29, 34], [116, 46, 44], [87, 30, 19], [154, 113, 93], [147, 117, 91], [96, 70, 43], [127, 95, 74], [140, 94, 81], [103, 39, 37], [135, 58, 64], [100, 14, 25], [130, 44, 53], [134, 55, 58], [105, 38, 32], [115, 62, 46], [99, 52, 32], [111, 68, 52], [71, 33, 20], [210, 181, 173], [255, 241, 237], [253, 248, 245], [252, 255, 255], [233, 242, 241], [248, 255, 255], [249, 255, 255], [252, 255, 251], [255, 255, 251], [255, 254, 251], [255, 253, 255], [255, 252, 255], [255, 252, 255], [255, 252, 255]], [[255, 254, 251], [237, 236, 234], [254, 255, 255], [251, 255, 255], [234, 241, 247], [251, 255, 255], [254, 255, 255], [248, 244, 241], [255, 250, 241], [255, 252, 239], [255, 250, 236], [255, 237, 223], [227, 208, 194], [196, 178, 166], [174, 160, 151], [165, 152, 144], [173, 150, 142], [168, 148, 139], [179, 159, 150], [204, 186, 176], [253, 234, 227], [255, 250, 243], [255, 250, 243], [255, 249, 243], [255, 247, 243], [180, 146, 144], [61, 21, 21], [70, 22, 22], [88, 33, 36], [102, 42, 44], [73, 7, 11], [76, 7, 12], [104, 29, 36], [97, 33, 34], [89, 39, 32], [224, 191, 176], [255, 237, 216], [242, 221, 202], [255, 247, 233], [254, 213, 209], [71, 12, 18], [82, 8, 23], [105, 22, 40], [109, 27, 41], [68, 0, 2], [110, 49, 44], [51, 4, 0], [78, 37, 15], [66, 25, 19], [145, 110, 106], [255, 237, 236], [255, 251, 251], [255, 253, 254], [251, 255, 255], [247, 255, 255], [234, 246, 242], [249, 255, 253], [252, 255, 250], [255, 255, 250], [255, 254, 251], [255, 252, 255], [255, 251, 255], [255, 251, 255], [255, 250, 255]], [[240, 237, 232], [255, 255, 251], [238, 240, 239], [248, 253, 255], [249, 255, 255], [249, 255, 255], [240, 245, 249], [255, 255, 255], [255, 250, 246], [255, 248, 242], [255, 248, 239], [255, 250, 241], [255, 252, 246], [255, 252, 250], [255, 253, 253], [254, 252, 253], [255, 247, 245], [255, 253, 251], [255, 254, 251], [255, 254, 251], [250, 246, 243], [250, 246, 243], [255, 254, 250], [248, 240, 237], [255, 242, 239], [255, 250, 246], [215, 190, 186], [195, 161, 159], [187, 146, 144], [255, 230, 230], [255, 207, 207], [176, 120, 121], [139, 68, 76], [104, 42, 45], [163, 115, 111], [255, 246, 233], [255, 250, 233], [255, 252, 236], [255, 249, 239], [255, 238, 237], [190, 134, 145], [103, 30, 49], [153, 71, 93], [177, 96, 113], [251, 180, 188], [255, 236, 232], [222, 179, 163], [179, 143, 121], [230, 192, 191], [255, 246, 244], [255, 246, 248], [255, 251, 253], [244, 244, 246], [245, 253, 255], [243, 255, 255], [247, 255, 255], [249, 255, 251], [252, 255, 250], [255, 255, 250], [255, 254, 251], [255, 252, 255], [255, 251, 255], [255, 250, 255], [255, 250, 255]]]]
nd_test = np.asarray(test)

inspect = tf.contrib.util.make_tensor_proto(nd_test, dtype='uint8', shape=nd_test.shape)

request.inputs['inputs'].CopyFrom(inspect)

start = int(round(time.time() * 1000))
tt, call = stub.Predict(request, 3, with_call=True)

With this script even for ONE SINGLE request, as you can see i set the predict timeout to 3s stub.Predict(request, 3, with_call=True), the client got the error:

Traceback (most recent call last):
  File "python2_insecure.py", line 37, in <module>
    tt, call = stub.Predict(request, 3, with_call=True)
  File "/usr/local/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 309, in __call__
    self._request_serializer, self._response_deserializer)
  File "/usr/local/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 195, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.ExpirationError: ExpirationError(code=StatusCode.DEADLINE_EXCEEDED, details="Deadline Exceeded")

But on TFS side, it shows (we just added logs to print the request duration):

2019-05-13 04:59:32.022336: I tensorflow_serving/model_servers/prediction_service_impl.cc:41] Clea Request Summary  ModelName: [inception] Success: [0] RequestDurationMillis: [6181]

So the server insists to fulfil the request by consuming more than 6s, though the request deadline is set to 3s.

This is just one single request case, for you to easily inspect the problem.

We have a professional gRPC attacker which we can set the attacking rate, attacking duration etc to allow us to do the proper performance test to all our gRPC servers. As i mentioned in previous post, when we send concurrent load to the TFS servers, the situation is getting a lot worse: the request context deadline is set to 10s, but we observed lots of requests last running in TFS server for more than 30 minutes, the full CPU resources was taken and never goes down in one hour period (i waited and observed for one hour), looks the only way to release the resources is forcedly kill the container.

lnshi on 13 May 2019

Thanks Inshi. Just as an update, we are able to re-produce the time-out part. Folks are on the way to collect necessary stack traces and doing further investigation.

minglotus-6 on 13 Jun 2019

👍1

Hi Inshi,
While the investigation on our side is still ongoing, could you use tools like lsstack or pstack to collect the abnormally hanging threads' stack traces (e.g: in the 30 minutes when requests were already seen as deadline exceeded)?
Those stack traces will help to narrow down where thread cancellation could go wrong.

minglotus-6 on 15 Jun 2019

Hi Inshi,

We haven't heard from you for 7+ days now. We will close this for now assuming you no longer need help on this issue. If you do need help, please feel free to reopen this issue.

Thanks

chanshah on 21 Jun 2019

😕1 👎1

I thought u guys can reproduce, anyway do whatever u want, close never mean
solved!

On Fri, 21 Jun 2019 at 23:56, chanshah notifications@github.com wrote:

Closed #1265 https://github.com/tensorflow/serving/issues/1265.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/serving/issues/1265?email_source=notifications&email_token=AF3RAEQI7PGMYAHJ6U2XU5DP3T2ZNA5CNFSM4GYTKLEKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOSDRFHOY#event-2430751675,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF3RAESGXDKTW7GIFF7NBIDP3T2ZNANCNFSM4GYTKLEA
.

lnshi on 21 Jun 2019

Hi Inshi, happy to reopen it.

As pointed by @minglotus-6, can you use tools like lsstack or pstack to collect the abnormally hanging threads' stack traces (e.g: in the 30 minutes when requests were already seen as deadline exceeded)?
Those stack traces will help to narrow down where thread cancellation could go wrong.

chanshah on 21 Jun 2019

Hi Inshi,
Sorry if I'm confusing you, we can re-produce the timed-out request.
However, this real issue here is that server wouldn't cancel threads efficiently (re lots of requests last running in TFS server for more than 30 minutes, the full CPU resources was taken and never goes down in one hour period) even after clients receive deadline exceeded rpc errors.
Could you collect and share the hanging threads' stack traces?

minglotus-6 on 28 Jun 2019

@lnshi ,
Can you please share the information requested by @minglotus-6 so that we can try resolving the issue. Thanks!

rmothukuru on 31 Jul 2019

i will try to find some time to do

lnshi on 31 Jul 2019

@rmothukuru meanwhile could u help to find some time to do also since i provided all the necessary setup, recently too busy

lnshi on 31 Jul 2019

_pstack 18009_

18009: tensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=inception --model_base_path=/inception/ (No symbols found in ) (No symbols found in /lib/x86_64-linux-gnu/libdl.so.2) (No symbols found in /lib/x86_64-linux-gnu/libm.so.6) (No symbols found in /lib/x86_64-linux-gnu/librt.so.1) (No symbols found in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) (No symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1) (No symbols found in /lib/x86_64-linux-gnu/libc.so.6) (No symbols found in /lib64/ld-linux-x86-64.so.2) 0x7f7475c9e4d9: _fini + 0x7f73ff076365 (7f747703d100, 7f747703d140, 672bb20, 0) 0x005690d2: AbslInternalPerThreadSemWait + 0x62 (672bb20, 3, 7ffe939bbb30, 0) 0x00567ffd: _ZN4absl5Mutex5BlockEPNS_13base_internal14PerThreadSynchE + 0x2d (36e0ca0, 7ffe939bbb30, 0, 0, 7f747703d100, 0) + 30 0x005689e5: _ZN4absl5Mutex11AwaitCommonERKNS_9ConditionENS_24synchronization_internal13KernelTimeoutE + 0x95 (672bb20, 672bae0) 0x00568a75: _ZN4absl5Mutex5AwaitERKNS_9ConditionE + 0x35 (533310, 533c30, 0, 0, 672bb28, 0) + 20 0x00533581: _ZN10tensorflow7serving8net_http12EvHTTPServer18WaitForTerminationEv + 0x51 (40e0f0, 7ffe939bbd70) 0x00508b6b: _ZN10tensorflow7serving4main6Server18WaitForTerminationEv + 0x1b (7ffe939bd2a8, 100000000, 0, 0, 0, 0) + 15e0 0x0041dc0b: main + 0xedb (11c00, 7ffe939bd2a8, 5939bd2d8, 41cd30, 0, 443dfc00600d1e63) + ffff80016fbba7b0 0x7f7475bbd830: _fini + 0x7f73fef956bc (42262d8d485500c8, 8949f689495300c8, 8ec8348e5294cd5, e966b7e803fdc148, db312074ed8548fc, 841f0f) + 41ff894152e9dd51 crawl: Input/output error Error tracing through process 18009

sreddybr3 on 1 Aug 2019

we need thread stacks for all threads of the modelserver to help debug background activity (after RPC is terminated/cancelled from the client side).

afaik there is no debian package for pstack and lsstack -- so installing these will be non-trivial (assuming they actually work). so please do not use these tools.

i'd recommend install elfutils package (apt install elfutils) on your system (where the modelserver is running) and then run /usr/bin/eu-stack -p <pid-of-modelserver> to dump thread stack of all threads.

ideally you want to collect stack traces (read: run eu-stack) after the RPC is done, but while the CPU usage is still high (on the server). this will help us see what (unnecessary) background activity is still happening.

netfs on 14 Aug 2019

👍1

Closing due to 14 days of inactivity - please do re-open if you manage to get stack trace for this :)

misterpeddy on 27 Aug 2019

👎2 😕1

#
@lnshi I know this query is completely out of context here. But I really want to know how you enabled request logs on TFS side. I did a lot of research to add the request logs on TFS side but i failed miserably. Can you please help me with this.

@minglotus-6 thanks a lot for you time.

Just now i tried and was able to reproduce it with this latest inception model: faster_rcnn_inception_resnet_v2_atrous_oidv4

model_config_list.conf

model_config_list: {
  config: {
    name: "inception",
    base_path: "/models/inception",
    model_platform: "tensorflow",
    model_version_policy: {
      specific: {
        versions: [1, 2, 3]
      }
    }
  }
}

Simple python script to send the gRPC inference request(input is just a small 2KB jpg image):

import grpc
from grpc._cython.cygrpc import CompressionAlgorithm
from grpc._cython.cygrpc import CompressionLevel

from grpc.beta import implementations
import tensorflow as tf
import os, time
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2

import numpy as np

MODEL_NAME = 'inception'
MODEL_SERVER_HOST = 'localhost'
MODEL_SERVER_PORT = 9000

opts = [('grpc.default_compression_algorithm', CompressionAlgorithm.gzip),
            ('grpc.grpc.default_compression_level', CompressionLevel.high)]

ch = grpc.insecure_channel(MODEL_SERVER_HOST + ":" + str(MODEL_SERVER_PORT), options=opts)
channel = implementations.Channel(ch)
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

request = predict_pb2.PredictRequest()
request.model_spec.name = MODEL_NAME
request.model_spec.version.value = 3
request.model_spec.signature_name = 'serving_default'

test = [[[[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [242, 255, 255], [248, 255, 255], [255, 253, 255], [255, 241, 235], [255, 244, 232], [254, 198, 181], [63, 0, 0], [150, 87, 70], [227, 171, 158], [255, 228, 219], [255, 246, 243], [255, 243, 243], [255, 250, 254], [249, 255, 255], [243, 255, 255], [239, 255, 255], [252, 252, 254], [254, 255, 255], [252, 255, 255], [251, 255, 255], [248, 255, 255], [247, 255, 255], [241, 255, 253], [238, 253, 250], [245, 255, 255], [249, 255, 255], [252, 255, 255], [255, 254, 255], [252, 246, 250], [255, 250, 255], [255, 246, 254], [255, 238, 247], [255, 232, 239], [255, 241, 234], [230, 183, 155], [151, 106, 65], [63, 19, 0], [219, 178, 156], [255, 244, 248], [255, 241, 255], [255, 247, 255], [255, 251, 255], [245, 249, 250], [250, 255, 243], [251, 255, 241], [252, 255, 246], [255, 254, 255], [255, 249, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [242, 255, 255], [249, 255, 255], [248, 239, 240], [255, 244, 237], [244, 202, 188], [62, 7, 0], [233, 171, 148], [202, 140, 119], [93, 36, 19], [174, 128, 115], [255, 223, 216], [255, 243, 242], [255, 252, 252], [252, 255, 255], [248, 255, 255], [245, 255, 255], [255, 255, 255], [255, 255, 255], [253, 254, 255], [247, 253, 253], [246, 255, 254], [247, 255, 255], [247, 255, 255], [247, 255, 255], [244, 254, 253], [242, 251, 250], [249, 253, 254], [251, 249, 252], [255, 252, 255], [251, 238, 245], [255, 249, 255], [255, 248, 255], [255, 238, 240], [164, 115, 101], [78, 29, 0], [192, 145, 101], [205, 162, 119], [73, 31, 7], [215, 179, 179], [255, 244, 255], [253, 235, 255], [255, 251, 255], [254, 255, 255], [251, 255, 244], [251, 255, 241], [251, 254, 245], [255, 254, 255], [251, 247, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [247, 255, 255], [250, 253, 255], [255, 252, 250], [255, 248, 237], [110, 67, 48], [182, 126, 99], [255, 196, 165], [255, 199, 171], [240, 183, 156], [121, 73, 53], [102, 66, 52], [255, 241, 233], [255, 252, 248], [237, 236, 234], [252, 255, 255], [251, 255, 255], [251, 249, 252], [254, 252, 255], [255, 255, 255], [252, 255, 255], [251, 255, 255], [249, 255, 255], [246, 255, 255], [244, 254, 253], [249, 255, 255], [251, 255, 255], [254, 255, 255], [248, 243, 247], [251, 242, 247], [255, 250, 255], [252, 235, 243], [253, 233, 242], [139, 86, 78], [126, 74, 53], [239, 190, 150], [249, 201, 152], [255, 214, 165], [191, 148, 116], [80, 41, 34], [251, 221, 233], [255, 242, 255], [255, 251, 255], [249, 251, 248], [252, 255, 243], [252, 255, 241], [249, 252, 243], [254, 253, 255], [254, 250, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [243, 250, 255], [250, 249, 254], [255, 252, 246], [156, 127, 111], [128, 82, 56], [255, 218, 183], [240, 177, 136], [246, 181, 141], [255, 200, 164], [255, 210, 179], [157, 116, 94], [81, 52, 38], [239, 221, 211], [255, 252, 248], [255, 254, 251], [255, 253, 254], [255, 253, 253], [255, 252, 252], [254, 250, 249], [250, 250, 248], [249, 251, 248], [250, 255, 251], [251, 255, 253], [251, 255, 253], [252, 255, 253], [241, 243, 240], [253, 252, 250], [255, 252, 253], [255, 248, 250], [253, 237, 240], [255, 248, 253], [109, 88, 93], [153, 98, 78], [255, 202, 171], [242, 190, 142], [233, 182, 125], [231, 182, 126], [245, 200, 161], [140, 98, 82], [159, 127, 130], [255, 247, 255], [255, 251, 255], [241, 240, 235], [252, 255, 243], [252, 255, 243], [247, 253, 243], [250, 249, 254], [255, 253, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [253, 253, 255], [252, 248, 249], [246, 229, 219], [63, 31, 8], [232, 186, 150], [229, 171, 125], [247, 181, 129], [255, 189, 137], [254, 192, 143], [224, 171, 129], [246, 201, 168], [164, 130, 105], [123, 95, 81], [251, 231, 222], [255, 250, 246], [253, 238, 235], [255, 252, 248], [255, 252, 248], [255, 253, 248], [255, 254, 248], [255, 254, 248], [255, 255, 248], [253, 253, 245], [251, 251, 243], [255, 255, 248], [255, 253, 247], [249, 240, 235], [255, 246, 243], [255, 250, 248], [255, 244, 243], [135, 111, 111], [122, 96, 97], [253, 197, 162], [236, 182, 136], [244, 189, 132], [239, 184, 119], [247, 193, 131], [255, 209, 162], [209, 165, 138], [67, 30, 22], [232, 208, 208], [255, 252, 250], [255, 252, 243], [251, 255, 240], [252, 255, 243], [249, 254, 247], [253, 254, 255], [255, 253, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 253, 255], [255, 252, 250], [175, 153, 139], [128, 93, 65], [244, 195, 152], [252, 193, 137], [246, 180, 119], [255, 203, 140], [243, 181, 120], [253, 196, 143], [251, 202, 159], [237, 196, 164], [110, 75, 53], [171, 142, 126], [255, 244, 233], [255, 248, 241], [255, 246, 235], [255, 245, 234], [255, 245, 233], [255, 247, 234], [255, 249, 236], [255, 253, 239], [255, 253, 239], [255, 253, 239], [247, 235, 221], [255, 252, 239], [255, 251, 239], [255, 243, 232], [255, 249, 239], [190, 161, 153], [66, 36, 28], [236, 203, 196], [255, 199, 149], [243, 187, 130], [255, 213, 146], [243, 185, 112], [241, 184, 115], [244, 189, 133], [255, 216, 178], [147, 109, 88], [149, 121, 110], [253, 236, 228], [255, 255, 243], [250, 252, 238], [247, 251, 237], [252, 255, 250], [254, 255, 255], [254, 254, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [247, 237, 238], [255, 241, 234], [96, 69, 50], [204, 166, 130], [244, 192, 142], [248, 187, 124], [255, 206, 135], [230, 162, 89], [234, 170, 99], [255, 201, 138], [249, 198, 145], [226, 178, 138], [222, 178, 149], [89, 48, 28], [212, 175, 159], [255, 245, 232], [251, 220, 199], [232, 201, 180], [200, 172, 150], [170, 144, 121], [151, 125, 102], [145, 121, 97], [147, 125, 101], [152, 130, 106], [192, 168, 144], [201, 175, 152], [229, 201, 179], [255, 238, 217], [245, 210, 190], [67, 29, 10], [228, 187, 169], [222, 181, 163], [234, 176, 112], [236, 177, 109], [238, 178, 105], [242, 180, 103], [255, 200, 127], [253, 195, 132], [234, 182, 134], [235, 192, 158], [102, 71, 50], [229, 210, 195], [255, 254, 241], [251, 251, 239], [249, 252, 243], [252, 255, 251], [252, 255, 255], [251, 255, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 251, 248], [216, 196, 185], [92, 62, 38], [240, 198, 158], [247, 195, 138], [250, 190, 120], [255, 188, 110], [240, 173, 92], [244, 178, 100], [225, 165, 95], [240, 186, 126], [255, 218, 169], [237, 189, 151], [179, 132, 104], [74, 28, 5], [152, 105, 85], [96, 53, 21], [115, 72, 40], [142, 101, 69], [169, 130, 97], [186, 147, 114], [188, 152, 118], [182, 146, 112], [177, 141, 107], [146, 110, 76], [133, 94, 61], [90, 49, 17], [149, 106, 74], [86, 41, 10], [149, 102, 72], [240, 191, 161], [255, 204, 175], [253, 197, 122], [244, 187, 110], [229, 168, 88], [226, 163, 84], [255, 193, 116], [239, 178, 111], [250, 195, 138], [243, 196, 152], [107, 72, 40], [209, 187, 164], [250, 238, 222], [253, 251, 239], [255, 255, 250], [249, 254, 250], [245, 249, 248], [252, 255, 255]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 250, 243], [171, 145, 130], [152, 116, 90], [255, 210, 168], [244, 190, 130], [241, 179, 106], [251, 184, 105], [236, 170, 86], [234, 169, 89], [237, 177, 104], [234, 179, 114], [202, 149, 95], [151, 98, 54], [126, 74, 37], [158, 105, 74], [207, 153, 125], [245, 192, 150], [246, 193, 151], [247, 196, 153], [250, 199, 156], [249, 200, 157], [249, 200, 157], [248, 199, 156], [248, 199, 156], [254, 205, 162], [242, 193, 150], [237, 186, 143], [221, 168, 126], [174, 119, 78], [132, 74, 34], [149, 91, 51], [198, 139, 99], [242, 187, 104], [220, 164, 81], [246, 185, 104], [243, 177, 99], [232, 166, 90], [251, 187, 116], [249, 191, 127], [255, 209, 156], [188, 151, 109], [136, 110, 83], [255, 250, 235], [255, 254, 248], [253, 253, 251], [254, 255, 255], [248, 252, 251], [252, 255, 253]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 240, 229], [113, 82, 64], [213, 174, 145], [230, 181, 138], [246, 190, 131], [255, 206, 133], [245, 180, 100], [234, 169, 87], [233, 170, 90], [242, 182, 109], [218, 161, 94], [174, 119, 62], [180, 127, 77], [235, 178, 135], [255, 205, 167], [253, 193, 157], [246, 184, 133], [247, 185, 134], [247, 187, 135], [249, 189, 137], [250, 190, 138], [248, 191, 138], [247, 190, 137], [246, 189, 136], [246, 189, 136], [242, 182, 130], [248, 188, 136], [255, 202, 151], [255, 200, 149], [234, 171, 120], [199, 136, 85], [179, 116, 65], [230, 178, 92], [238, 183, 100], [237, 176, 96], [232, 166, 92], [255, 189, 116], [253, 186, 115], [255, 199, 132], [239, 187, 127], [215, 175, 126], [110, 81, 49], [254, 237, 221], [255, 252, 250], [248, 246, 249], [254, 255, 255], [246, 251, 247], [252, 255, 251]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [250, 218, 203], [95, 60, 38], [234, 191, 159], [244, 193, 148], [253, 194, 136], [233, 170, 101], [243, 177, 101], [237, 174, 95], [238, 176, 99], [246, 188, 115], [245, 190, 125], [240, 186, 126], [247, 192, 138], [255, 200, 152], [253, 192, 147], [233, 172, 127], [251, 186, 128], [252, 187, 129], [252, 187, 129], [253, 188, 130], [253, 188, 130], [253, 188, 130], [250, 188, 129], [249, 187, 128], [255, 196, 137], [255, 192, 134], [251, 186, 128], [250, 185, 127], [255, 195, 137], [255, 201, 143], [255, 189, 131], [235, 169, 111], [235, 188, 100], [245, 193, 110], [227, 169, 95], [232, 167, 99], [247, 178, 111], [242, 173, 104], [243, 180, 109], [255, 202, 136], [242, 201, 147], [89, 59, 23], [234, 215, 200], [255, 251, 253], [247, 242, 249], [254, 255, 255], [245, 250, 246], [252, 255, 248]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [231, 196, 176], [116, 79, 53], [231, 186, 153], [252, 199, 155], [251, 192, 136], [229, 168, 103], [243, 180, 109], [240, 178, 105], [249, 190, 120], [239, 184, 117], [246, 192, 128], [255, 208, 148], [252, 197, 141], [231, 174, 119], [235, 175, 121], [255, 197, 143], [255, 191, 130], [255, 191, 130], [255, 192, 130], [255, 192, 130], [255, 191, 129], [255, 191, 129], [254, 190, 128], [254, 190, 128], [246, 182, 120], [255, 193, 131], [255, 194, 132], [248, 184, 122], [245, 181, 119], [254, 190, 128], [255, 196, 134], [255, 195, 133], [243, 199, 112], [247, 198, 119], [239, 182, 115], [243, 179, 118], [224, 155, 96], [255, 193, 128], [238, 175, 104], [255, 201, 131], [251, 209, 151], [105, 73, 35], [214, 190, 178], [255, 250, 255], [253, 245, 255], [255, 254, 255], [247, 252, 248], [252, 255, 246]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [221, 183, 160], [127, 86, 58], [253, 207, 173], [233, 180, 136], [249, 189, 137], [255, 196, 137], [235, 174, 111], [240, 180, 117], [248, 193, 129], [246, 195, 132], [239, 189, 130], [230, 180, 121], [233, 181, 123], [249, 193, 134], [255, 200, 140], [255, 200, 138], [253, 192, 129], [252, 191, 128], [252, 191, 128], [251, 190, 127], [250, 189, 126], [249, 188, 125], [249, 188, 125], [249, 188, 125], [251, 190, 127], [249, 188, 125], [248, 187, 124], [251, 190, 127], [252, 192, 129], [252, 192, 129], [250, 190, 127], [248, 188, 125], [235, 194, 112], [253, 206, 134], [251, 197, 137], [248, 185, 134], [234, 164, 113], [255, 191, 130], [255, 206, 138], [228, 174, 102], [247, 202, 143], [143, 109, 72], [198, 171, 162], [255, 243, 253], [255, 250, 255], [251, 250, 255], [253, 255, 252], [252, 255, 244]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [213, 171, 146], [138, 94, 65], [255, 207, 171], [249, 194, 153], [239, 178, 131], [182, 121, 67], [225, 164, 109], [247, 188, 132], [243, 188, 131], [252, 201, 144], [252, 203, 147], [242, 193, 135], [242, 192, 131], [253, 199, 135], [249, 192, 125], [233, 174, 106], [249, 194, 130], [248, 193, 129], [247, 192, 128], [248, 190, 127], [247, 189, 126], [247, 189, 126], [248, 190, 127], [248, 190, 127], [255, 201, 138], [248, 190, 127], [241, 186, 122], [245, 190, 126], [245, 191, 127], [242, 188, 124], [245, 191, 127], [253, 199, 135], [233, 198, 117], [242, 199, 131], [238, 187, 134], [254, 193, 149], [250, 182, 137], [173, 104, 49], [255, 194, 126], [248, 191, 120], [244, 198, 138], [163, 127, 91], [188, 161, 152], [255, 242, 255], [255, 249, 255], [248, 245, 255], [254, 255, 251], [251, 255, 242]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [214, 170, 143], [147, 102, 71], [236, 186, 151], [255, 201, 160], [174, 116, 70], [131, 69, 20], [236, 176, 124], [250, 192, 142], [245, 192, 140], [239, 189, 138], [238, 191, 139], [241, 194, 138], [239, 191, 129], [235, 184, 118], [245, 191, 121], [255, 204, 131], [248, 196, 136], [247, 195, 135], [246, 194, 134], [245, 193, 133], [244, 192, 132], [247, 193, 133], [248, 194, 134], [249, 195, 135], [233, 181, 121], [240, 188, 128], [251, 199, 139], [255, 206, 145], [249, 199, 138], [237, 189, 127], [236, 188, 126], [241, 193, 131], [240, 206, 132], [230, 191, 126], [231, 180, 133], [253, 193, 156], [252, 184, 145], [151, 81, 30], [178, 113, 47], [249, 192, 121], [254, 208, 148], [148, 112, 76], [187, 158, 152], [255, 247, 255], [255, 249, 255], [251, 247, 255], [254, 255, 251], [249, 255, 240]], [[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [224, 180, 151], [139, 92, 62], [238, 188, 153], [214, 159, 120], [88, 29, 0], [209, 148, 101], [255, 202, 155], [245, 189, 140], [246, 194, 146], [239, 191, 142], [236, 190, 140], [239, 194, 137], [246, 198, 136], [247, 196, 130], [246, 192, 120], [246, 190, 115], [243, 194, 135], [243, 194, 135], [241, 192, 133], [241, 191, 132], [241, 191, 132], [242, 192, 133], [244, 194, 135], [245, 195, 136], [246, 196, 137], [246, 196, 137], [243, 194, 135], [238, 192, 132], [239, 193, 133], [242, 198, 137], [242, 198, 137], [239, 195, 134], [236, 204, 131], [237, 199, 137], [243, 195, 149], [245, 185, 151], [252, 184, 149], [240, 172, 123], [107, 43, 0], [187, 130, 59], [255, 220, 160], [125, 87, 51], [187, 158, 152], [255, 247, 255], [253, 241, 255], [255, 252, 255], [254, 255, 251], [248, 255, 237]], [[249, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [251, 249, 250], [254, 252, 253], [255, 254, 255], [255, 254, 255], [255, 253, 254], [253, 251, 252], [255, 253, 254], [255, 255, 255], [229, 189, 187], [124, 84, 74], [234, 190, 165], [71, 24, 0], [208, 156, 98], [255, 205, 136], [250, 187, 116], [253, 184, 117], [255, 187, 128], [255, 185, 137], [255, 185, 145], [255, 187, 150], [252, 190, 151], [248, 191, 146], [243, 191, 141], [240, 190, 137], [249, 195, 109], [249, 194, 112], [249, 193, 119], [249, 191, 128], [249, 189, 135], [249, 188, 141], [249, 188, 141], [249, 189, 139], [245, 186, 130], [253, 195, 132], [255, 199, 130], [251, 195, 120], [250, 194, 117], [253, 197, 120], [252, 196, 119], [247, 191, 116], [252, 190, 129], [252, 190, 129], [255, 196, 133], [255, 191, 127], [241, 180, 117], [255, 198, 141], [220, 159, 112], [84, 28, 0], [235, 185, 162], [139, 95, 84], [207, 172, 170], [255, 247, 250], [255, 245, 248], [255, 248, 251], [255, 254, 255], [251, 253, 252]], [[251, 255, 255], [251, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [254, 252, 253], [255, 254, 255], [255, 254, 255], [255, 254, 255], [254, 252, 253], [253, 251, 252], [253, 253, 253], [255, 255, 255], [244, 204, 202], [116, 73, 64], [114, 70, 45], [171, 124, 82], [247, 197, 138], [242, 187, 120], [246, 186, 116], [255, 192, 126], [255, 197, 138], [255, 189, 140], [247, 180, 137], [244, 179, 139], [244, 185, 141], [245, 192, 142], [244, 193, 138], [240, 191, 132], [247, 195, 111], [247, 194, 114], [247, 193, 121], [247, 192, 128], [247, 190, 135], [247, 189, 141], [247, 189, 141], [247, 189, 139], [251, 195, 138], [250, 195, 131], [246, 192, 122], [242, 188, 114], [243, 189, 115], [246, 192, 118], [247, 193, 121], [245, 191, 119], [248, 189, 129], [245, 187, 124], [248, 188, 125], [253, 194, 128], [245, 185, 122], [255, 198, 141], [248, 190, 140], [171, 115, 78], [131, 81, 56], [99, 55, 42], [230, 197, 192], [255, 248, 248], [255, 248, 251], [255, 254, 255], [254, 255, 255], [251, 255, 255]], [[252, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [253, 253, 253], [252, 252, 252], [254, 254, 254], [255, 255, 255], [255, 239, 238], [137, 94, 85], [86, 42, 17], [230, 185, 143], [247, 198, 140], [236, 185, 119], [255, 199, 130], [250, 191, 125], [247, 185, 126], [252, 190, 139], [255, 195, 148], [253, 195, 149], [246, 190, 139], [239, 189, 130], [240, 192, 126], [242, 198, 127], [246, 195, 116], [246, 194, 119], [246, 193, 125], [246, 192, 132], [246, 191, 137], [246, 190, 141], [246, 190, 141], [246, 191, 137], [250, 195, 138], [244, 190, 126], [241, 188, 120], [244, 191, 121], [248, 195, 125], [249, 196, 128], [249, 195, 131], [251, 197, 133], [246, 194, 136], [244, 192, 132], [242, 188, 124], [253, 198, 133], [248, 191, 124], [245, 187, 124], [255, 200, 145], [240, 187, 143], [123, 74, 42], [108, 67, 49], [255, 238, 229], [255, 249, 246], [255, 246, 247], [253, 253, 255], [249, 255, 255], [247, 255, 255]], [[254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [254, 254, 254], [254, 254, 254], [254, 254, 254], [255, 255, 255], [255, 255, 255], [255, 241, 237], [134, 88, 75], [173, 127, 101], [238, 191, 149], [236, 189, 133], [240, 191, 125], [248, 197, 131], [244, 190, 126], [238, 183, 126], [243, 188, 134], [249, 193, 144], [248, 195, 143], [244, 194, 135], [241, 194, 126], [239, 196, 118], [241, 199, 117], [243, 195, 119], [243, 195, 123], [243, 194, 128], [243, 193, 134], [243, 192, 137], [243, 192, 139], [243, 192, 139], [243, 192, 135], [243, 193, 132], [240, 191, 125], [242, 193, 126], [249, 200, 133], [249, 200, 134], [244, 194, 133], [242, 191, 134], [245, 194, 139], [238, 193, 138], [244, 198, 139], [239, 191, 127], [249, 198, 132], [248, 195, 127], [234, 179, 114], [247, 191, 132], [255, 202, 152], [194, 146, 108], [125, 85, 60], [255, 241, 227], [255, 250, 243], [254, 249, 246], [251, 255, 255], [245, 255, 255], [238, 255, 255]], [[255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [254, 254, 254], [254, 254, 254], [253, 253, 253], [254, 254, 254], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 205, 197], [106, 55, 38], [233, 184, 154], [244, 197, 153], [253, 206, 150], [237, 192, 127], [228, 180, 114], [246, 198, 134], [250, 201, 143], [240, 190, 139], [231, 181, 130], [232, 182, 129], [241, 193, 131], [247, 200, 128], [244, 199, 116], [239, 195, 106], [242, 195, 125], [242, 195, 127], [242, 194, 130], [242, 193, 134], [242, 193, 137], [242, 193, 137], [242, 193, 135], [242, 194, 132], [242, 194, 128], [242, 195, 127], [245, 198, 130], [246, 198, 132], [242, 193, 134], [236, 186, 133], [236, 185, 138], [240, 189, 144], [226, 186, 134], [242, 201, 145], [233, 189, 128], [239, 191, 125], [249, 196, 126], [241, 187, 117], [247, 192, 127], [248, 196, 139], [227, 178, 135], [111, 70, 40], [216, 188, 167], [255, 251, 241], [255, 255, 250], [249, 255, 255], [242, 255, 255], [231, 252, 253]], [[255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [253, 253, 253], [251, 253, 252], [251, 253, 252], [253, 255, 254], [254, 255, 255], [254, 255, 255], [254, 255, 255], [252, 254, 253], [205, 148, 129], [145, 89, 64], [242, 188, 152], [240, 190, 141], [250, 201, 142], [243, 198, 131], [236, 191, 126], [244, 198, 138], [238, 191, 137], [243, 195, 147], [249, 201, 153], [251, 201, 150], [246, 198, 136], [241, 194, 122], [241, 195, 110], [243, 198, 107], [242, 195, 127], [242, 194, 128], [242, 194, 132], [242, 193, 134], [242, 193, 135], [242, 193, 135], [242, 194, 132], [242, 194, 128], [237, 190, 120], [242, 195, 125], [243, 196, 126], [240, 192, 128], [240, 191, 133], [245, 194, 147], [249, 197, 157], [250, 198, 161], [228, 192, 144], [247, 207, 156], [234, 192, 134], [235, 187, 121], [245, 192, 122], [249, 193, 119], [253, 196, 127], [241, 185, 124], [253, 201, 153], [141, 98, 63], [163, 131, 108], [255, 251, 239], [255, 255, 250], [241, 250, 249], [242, 255, 255], [238, 255, 255]], [[255, 252, 255], [255, 253, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [253, 255, 254], [252, 254, 253], [252, 254, 253], [254, 255, 255], [254, 255, 255], [254, 254, 254], [250, 250, 250], [246, 246, 246], [161, 98, 67], [215, 152, 117], [253, 195, 149], [231, 176, 119], [239, 188, 123], [249, 200, 133], [250, 202, 138], [240, 194, 135], [231, 183, 134], [237, 188, 145], [244, 192, 152], [244, 193, 148], [241, 190, 135], [240, 189, 123], [242, 193, 116], [248, 197, 114], [243, 194, 128], [243, 193, 130], [243, 193, 132], [243, 193, 132], [243, 193, 134], [243, 193, 132], [243, 194, 128], [243, 194, 125], [235, 187, 113], [243, 195, 121], [244, 196, 124], [239, 190, 124], [244, 193, 138], [253, 200, 156], [247, 193, 157], [231, 176, 146], [223, 184, 143], [242, 201, 155], [240, 195, 140], [242, 192, 129], [244, 190, 120], [248, 188, 115], [255, 196, 124], [240, 180, 117], [255, 212, 158], [212, 164, 126], [123, 87, 61], [251, 229, 215], [255, 254, 248], [242, 248, 248], [243, 255, 255], [240, 255, 255]], [[255, 251, 255], [255, 252, 255], [255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [253, 255, 254], [254, 255, 255], [255, 255, 255], [253, 253, 253], [247, 245, 246], [241, 239, 240], [125, 58, 16], [225, 161, 113], [255, 206, 152], [242, 184, 120], [255, 204, 134], [244, 191, 121], [235, 186, 120], [245, 194, 139], [255, 210, 165], [224, 170, 134], [187, 132, 101], [189, 133, 98], [223, 166, 123], [252, 195, 140], [253, 198, 133], [244, 187, 116], [246, 192, 128], [246, 192, 130], [246, 192, 130], [246, 192, 132], [246, 192, 132], [246, 192, 128], [246, 193, 125], [246, 194, 121], [242, 190, 115], [252, 200, 125], [251, 199, 126], [243, 189, 125], [245, 190, 136], [246, 188, 148], [220, 161, 129], [183, 123, 97], [204, 160, 125], [224, 179, 138], [239, 189, 140], [252, 198, 138], [248, 189, 121], [249, 185, 113], [255, 198, 127], [247, 183, 119], [240, 180, 126], [246, 194, 154], [88, 48, 22], [232, 204, 190], [255, 252, 246], [255, 255, 255], [238, 247, 252], [242, 255, 255]], [[255, 251, 255], [255, 251, 255], [255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [249, 249, 249], [252, 250, 251], [250, 248, 249], [255, 254, 255], [195, 191, 192], [140, 73, 21], [255, 191, 136], [255, 193, 129], [255, 196, 124], [244, 186, 112], [244, 188, 114], [249, 194, 129], [251, 196, 142], [248, 190, 152], [195, 135, 107], [204, 144, 120], [130, 68, 45], [102, 39, 8], [233, 171, 130], [252, 190, 141], [255, 195, 140], [248, 193, 129], [252, 196, 135], [233, 177, 116], [255, 201, 142], [255, 203, 144], [226, 171, 107], [252, 197, 130], [248, 194, 122], [243, 189, 115], [255, 203, 129], [231, 177, 105], [245, 190, 126], [243, 186, 133], [243, 184, 144], [206, 145, 114], [182, 120, 95], [64, 13, 0], [174, 122, 85], [248, 194, 148], [247, 188, 132], [253, 189, 125], [245, 178, 108], [255, 197, 128], [255, 188, 125], [254, 191, 140], [251, 196, 157], [134, 90, 65], [215, 183, 170], [255, 243, 239], [250, 241, 244], [254, 255, 255], [248, 252, 255]], [[255, 251, 255], [255, 252, 255], [255, 252, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [247, 249, 248], [255, 255, 255], [255, 254, 255], [255, 254, 255], [253, 247, 249], [255, 252, 255], [175, 166, 169], [179, 114, 58], [255, 202, 141], [248, 187, 120], [249, 189, 116], [249, 190, 114], [250, 192, 118], [250, 190, 127], [246, 184, 133], [235, 172, 137], [192, 127, 105], [255, 217, 201], [182, 116, 102], [71, 6, 0], [192, 128, 100], [255, 215, 179], [231, 168, 125], [246, 190, 129], [251, 195, 136], [255, 209, 152], [231, 175, 118], [239, 183, 126], [253, 197, 138], [251, 195, 134], [239, 184, 119], [252, 198, 128], [248, 194, 124], [255, 204, 137], [254, 198, 137], [255, 197, 147], [200, 141, 101], [215, 154, 123], [255, 218, 193], [73, 16, 0], [116, 57, 25], [235, 173, 132], [255, 197, 144], [255, 193, 133], [255, 191, 126], [255, 188, 123], [250, 184, 124], [247, 185, 136], [255, 202, 167], [171, 125, 102], [183, 147, 135], [255, 247, 246], [255, 250, 253], [255, 250, 255], [255, 253, 255]], [[255, 252, 255], [255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [244, 246, 245], [255, 255, 255], [255, 254, 255], [251, 245, 247], [255, 251, 255], [255, 250, 254], [169, 154, 159], [159, 102, 49], [238, 182, 125], [241, 186, 121], [253, 196, 125], [255, 197, 123], [249, 189, 117], [253, 192, 129], [255, 196, 145], [216, 150, 116], [127, 59, 38], [143, 75, 64], [103, 35, 26], [73, 7, 0], [186, 124, 103], [248, 187, 159], [248, 189, 155], [248, 193, 136], [234, 179, 123], [246, 191, 137], [250, 194, 145], [255, 205, 156], [240, 184, 135], [245, 189, 138], [254, 199, 143], [234, 180, 120], [255, 204, 142], [241, 187, 127], [214, 159, 103], [255, 215, 168], [183, 125, 87], [91, 32, 2], [151, 91, 65], [106, 42, 17], [80, 15, 0], [223, 158, 120], [255, 194, 146], [247, 182, 124], [255, 194, 133], [247, 183, 122], [252, 191, 136], [237, 180, 135], [255, 205, 173], [158, 113, 92], [153, 116, 107], [255, 247, 244], [255, 246, 248], [255, 240, 245], [255, 251, 255]], [[255, 252, 255], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [248, 252, 251], [254, 255, 255], [235, 233, 234], [255, 253, 255], [255, 251, 255], [255, 243, 248], [255, 249, 255], [117, 98, 104], [192, 147, 105], [243, 197, 148], [242, 195, 139], [248, 199, 133], [251, 198, 128], [239, 182, 113], [243, 181, 120], [254, 188, 138], [238, 170, 135], [95, 27, 6], [77, 9, 0], [129, 65, 55], [70, 8, 0], [190, 133, 114], [237, 183, 157], [242, 191, 160], [255, 206, 155], [243, 193, 144], [243, 192, 147], [221, 169, 129], [153, 101, 64], [135, 83, 46], [147, 95, 56], [227, 176, 133], [248, 197, 150], [243, 193, 144], [244, 194, 145], [248, 197, 150], [240, 189, 146], [198, 145, 111], [71, 18, 0], [62, 8, 0], [107, 40, 14], [93, 26, 0], [226, 161, 123], [255, 191, 145], [243, 182, 128], [253, 194, 136], [243, 187, 130], [249, 197, 147], [248, 199, 158], [231, 187, 158], [216, 178, 159], [109, 76, 67], [255, 236, 233], [255, 243, 245], [255, 249, 253], [255, 250, 253]], [[255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [246, 246, 246], [246, 242, 243], [255, 252, 255], [255, 250, 255], [255, 236, 242], [185, 164, 171], [110, 87, 95], [255, 230, 202], [255, 222, 189], [226, 189, 145], [221, 180, 126], [245, 199, 137], [246, 195, 130], [253, 194, 134], [254, 190, 142], [247, 181, 146], [162, 96, 70], [97, 31, 15], [106, 47, 33], [78, 25, 7], [202, 156, 132], [230, 189, 157], [231, 193, 157], [213, 172, 126], [251, 209, 167], [255, 215, 177], [230, 187, 155], [61, 17, 0], [86, 41, 18], [70, 25, 2], [207, 163, 136], [255, 216, 184], [247, 204, 169], [220, 178, 138], [219, 177, 139], [241, 198, 163], [209, 166, 134], [87, 43, 16], [106, 62, 37], [85, 15, 0], [142, 75, 46], [241, 177, 139], [254, 193, 148], [253, 197, 146], [241, 192, 136], [237, 191, 139], [231, 191, 142], [233, 195, 158], [225, 191, 164], [255, 237, 219], [147, 120, 111], [156, 132, 128], [255, 249, 248], [255, 237, 238], [255, 250, 250]], [[255, 254, 255], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [237, 237, 237], [255, 253, 255], [255, 251, 255], [249, 232, 238], [199, 178, 185], [122, 96, 105], [214, 187, 196], [255, 241, 231], [242, 223, 206], [227, 202, 172], [217, 185, 144], [241, 201, 149], [241, 195, 136], [249, 194, 137], [251, 191, 141], [246, 183, 142], [217, 153, 125], [117, 59, 35], [74, 22, 0], [152, 108, 83], [228, 192, 160], [237, 208, 168], [247, 223, 179], [255, 245, 207], [243, 209, 174], [247, 212, 184], [255, 218, 199], [238, 200, 187], [66, 28, 19], [188, 150, 141], [255, 234, 223], [249, 212, 194], [234, 198, 174], [255, 248, 220], [255, 228, 198], [243, 208, 180], [214, 178, 152], [154, 118, 94], [76, 40, 18], [126, 59, 32], [213, 148, 116], [251, 191, 154], [238, 184, 138], [243, 196, 144], [221, 181, 129], [230, 197, 146], [221, 192, 148], [227, 203, 167], [247, 225, 201], [253, 232, 215], [241, 221, 212], [108, 89, 83], [183, 164, 160], [249, 230, 226], [255, 250, 246]], [[255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [249, 255, 255], [249, 255, 255], [251, 255, 255], [238, 242, 241], [255, 255, 255], [242, 236, 238], [255, 251, 255], [246, 227, 233], [121, 95, 104], [120, 90, 100], [255, 231, 242], [247, 240, 247], [246, 236, 234], [255, 243, 227], [242, 219, 187], [247, 214, 169], [231, 190, 136], [234, 183, 128], [234, 178, 127], [255, 202, 158], [235, 176, 142], [228, 175, 144], [226, 181, 152], [232, 196, 162], [240, 215, 174], [255, 252, 204], [249, 236, 183], [255, 241, 208], [206, 179, 150], [71, 43, 22], [186, 156, 145], [163, 132, 129], [63, 31, 34], [144, 112, 115], [195, 163, 164], [64, 33, 28], [178, 148, 137], [255, 237, 219], [247, 219, 197], [255, 230, 208], [250, 222, 200], [232, 204, 182], [195, 167, 146], [218, 151, 122], [255, 197, 162], [255, 198, 158], [231, 183, 137], [240, 200, 149], [221, 189, 138], [241, 218, 168], [237, 218, 176], [255, 243, 210], [252, 240, 216], [255, 250, 234], [251, 238, 229], [132, 117, 110], [82, 67, 60], [255, 251, 244], [255, 240, 232]], [[255, 255, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [251, 255, 255], [249, 255, 255], [249, 255, 255], [251, 255, 255], [252, 255, 255], [251, 251, 251], [255, 253, 255], [222, 210, 214], [231, 210, 217], [168, 142, 151], [34, 4, 14], [224, 192, 203], [250, 249, 255], [242, 240, 245], [255, 250, 241], [247, 230, 204], [255, 241, 199], [255, 228, 177], [255, 222, 168], [255, 210, 158], [247, 190, 145], [255, 217, 180], [249, 199, 166], [255, 221, 187], [255, 232, 194], [255, 241, 196], [251, 240, 186], [255, 247, 190], [255, 247, 216], [247, 223, 197], [223, 197, 182], [164, 137, 130], [195, 166, 168], [229, 199, 207], [211, 181, 191], [160, 131, 136], [206, 177, 179], [255, 234, 227], [255, 238, 225], [255, 242, 225], [253, 228, 208], [255, 244, 224], [255, 233, 213], [246, 221, 201], [255, 204, 172], [255, 199, 164], [255, 200, 159], [247, 200, 154], [255, 225, 173], [248, 220, 170], [255, 242, 194], [255, 244, 202], [254, 245, 212], [244, 236, 213], [249, 242, 226], [224, 214, 204], [35, 25, 16], [129, 114, 107], [211, 197, 188], [233, 216, 206]], [[251, 255, 253], [251, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 254, 251], [244, 234, 232], [255, 251, 248], [147, 128, 122], [30, 7, 1], [190, 163, 154], [235, 207, 196], [255, 230, 220], [238, 251, 231], [239, 252, 234], [225, 239, 224], [223, 239, 228], [233, 250, 242], [230, 249, 247], [228, 246, 248], [232, 247, 252], [234, 245, 251], [236, 243, 249], [241, 242, 247], [246, 240, 244], [250, 238, 240], [255, 237, 237], [255, 235, 233], [255, 234, 233], [245, 246, 241], [246, 247, 242], [247, 248, 243], [249, 248, 244], [250, 249, 244], [251, 250, 245], [252, 251, 246], [253, 253, 245], [243, 243, 235], [243, 243, 233], [243, 243, 233], [242, 244, 233], [241, 245, 231], [241, 245, 231], [241, 245, 231], [239, 245, 231], [234, 240, 230], [251, 255, 247], [243, 249, 239], [231, 234, 225], [243, 244, 236], [246, 246, 236], [243, 240, 231], [254, 247, 239], [249, 236, 228], [245, 228, 221], [255, 236, 232], [255, 239, 237], [239, 205, 203], [214, 176, 175], [57, 15, 16], [113, 69, 70]], [[252, 255, 253], [252, 255, 253], [252, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [255, 253, 253], [249, 239, 238], [255, 241, 238], [249, 231, 227], [56, 36, 29], [208, 183, 176], [255, 248, 239], [255, 242, 234], [229, 233, 216], [251, 255, 242], [239, 247, 232], [238, 248, 237], [247, 255, 250], [225, 237, 233], [222, 234, 234], [247, 255, 255], [236, 244, 247], [238, 243, 246], [242, 242, 244], [245, 241, 242], [249, 239, 238], [253, 238, 235], [255, 237, 233], [255, 237, 231], [239, 240, 235], [239, 240, 235], [241, 240, 235], [242, 241, 236], [242, 241, 236], [243, 242, 237], [243, 243, 235], [244, 244, 236], [243, 243, 235], [243, 243, 233], [242, 244, 233], [242, 244, 233], [241, 244, 233], [241, 245, 231], [239, 245, 231], [239, 245, 231], [239, 242, 233], [228, 231, 222], [237, 240, 231], [245, 246, 238], [251, 253, 242], [237, 237, 227], [240, 236, 227], [237, 230, 222], [242, 231, 225], [255, 251, 244], [255, 240, 235], [255, 237, 234], [255, 246, 244], [213, 177, 177], [68, 29, 30], [246, 206, 207]], [[254, 255, 253], [254, 255, 253], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [254, 250, 249], [255, 253, 251], [251, 241, 239], [255, 247, 244], [241, 223, 219], [72, 51, 46], [128, 105, 99], [154, 131, 123], [200, 194, 180], [231, 225, 213], [255, 255, 244], [249, 246, 237], [238, 238, 230], [247, 248, 242], [238, 240, 235], [230, 232, 229], [241, 243, 240], [242, 242, 240], [243, 242, 238], [245, 242, 237], [246, 241, 235], [248, 241, 233], [249, 240, 231], [250, 240, 231], [242, 241, 236], [242, 241, 236], [242, 241, 236], [242, 241, 236], [242, 242, 234], [242, 242, 234], [242, 242, 234], [242, 242, 234], [243, 243, 235], [242, 244, 233], [242, 244, 233], [241, 244, 233], [241, 244, 233], [239, 245, 233], [239, 245, 233], [239, 245, 233], [234, 234, 224], [252, 252, 242], [250, 250, 240], [221, 221, 211], [252, 249, 240], [250, 247, 238], [253, 249, 240], [251, 244, 236], [249, 238, 232], [255, 247, 242], [199, 180, 176], [156, 132, 130], [139, 111, 110], [76, 44, 45], [236, 202, 203], [255, 232, 234]], [[255, 255, 253], [255, 255, 253], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [248, 246, 247], [255, 251, 252], [250, 241, 242], [255, 252, 250], [255, 250, 247], [255, 247, 242], [114, 96, 92], [53, 36, 29], [229, 210, 203], [255, 251, 243], [230, 213, 205], [247, 232, 225], [251, 238, 230], [232, 221, 215], [255, 253, 246], [238, 231, 223], [246, 242, 233], [245, 242, 233], [245, 242, 233], [243, 243, 233], [243, 243, 233], [243, 243, 231], [243, 243, 231], [243, 243, 231], [249, 246, 239], [248, 245, 238], [248, 245, 238], [246, 246, 238], [245, 245, 237], [245, 245, 237], [244, 244, 236], [244, 244, 236], [242, 243, 235], [242, 243, 235], [241, 244, 235], [241, 244, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [255, 248, 238], [241, 237, 226], [248, 244, 233], [242, 238, 229], [255, 255, 246], [230, 226, 217], [240, 233, 225], [252, 245, 237], [242, 233, 226], [255, 245, 242], [245, 230, 227], [57, 37, 36], [55, 31, 31], [255, 230, 231], [255, 243, 245], [255, 246, 248]], [[255, 254, 251], [255, 254, 251], [255, 255, 253], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [251, 249, 252], [255, 254, 255], [240, 234, 234], [250, 242, 240], [255, 252, 250], [232, 222, 220], [38, 27, 23], [190, 165, 161], [240, 215, 211], [255, 242, 238], [255, 249, 243], [255, 249, 243], [250, 230, 221], [236, 218, 208], [255, 248, 237], [252, 240, 228], [249, 241, 228], [246, 242, 230], [243, 244, 230], [241, 245, 231], [239, 245, 231], [238, 246, 231], [238, 246, 233], [248, 244, 235], [247, 243, 234], [246, 243, 234], [245, 242, 233], [244, 241, 232], [242, 242, 232], [241, 241, 231], [240, 242, 231], [242, 244, 233], [241, 244, 235], [241, 244, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [255, 251, 242], [238, 228, 219], [250, 240, 231], [245, 236, 227], [247, 238, 229], [249, 242, 234], [255, 254, 246], [243, 236, 230], [255, 253, 248], [224, 215, 210], [180, 169, 167], [75, 59, 59], [189, 171, 171], [255, 247, 249], [255, 247, 249], [255, 245, 248]], [[255, 253, 251], [255, 254, 251], [255, 253, 253], [255, 254, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [251, 252, 255], [250, 251, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [254, 250, 249], [243, 239, 238], [245, 240, 237], [139, 113, 116], [90, 64, 65], [150, 124, 123], [166, 141, 137], [187, 162, 157], [255, 241, 234], [255, 244, 233], [251, 232, 218], [255, 239, 224], [252, 240, 224], [248, 242, 226], [243, 244, 228], [241, 245, 231], [238, 246, 233], [235, 247, 235], [234, 248, 235], [249, 242, 232], [247, 243, 232], [246, 242, 231], [246, 242, 231], [244, 242, 230], [242, 242, 230], [241, 241, 231], [240, 242, 231], [241, 244, 233], [241, 244, 235], [241, 244, 235], [239, 245, 235], [239, 245, 235], [239, 245, 235], [239, 244, 237], [239, 244, 237], [244, 227, 219], [255, 251, 243], [253, 239, 230], [248, 235, 227], [255, 253, 244], [235, 226, 219], [182, 173, 166], [184, 177, 171], [139, 131, 128], [89, 81, 78], [119, 111, 109], [240, 230, 229], [255, 244, 246], [255, 251, 253], [253, 238, 243], [255, 250, 255]], [[255, 252, 250], [255, 253, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [229, 233, 236], [254, 255, 255], [250, 251, 253], [244, 244, 244], [255, 255, 255], [249, 249, 249], [255, 248, 254], [250, 229, 234], [202, 182, 184], [184, 163, 162], [113, 89, 85], [116, 91, 84], [245, 223, 212], [255, 245, 232], [255, 238, 224], [253, 240, 224], [249, 242, 226], [246, 242, 230], [242, 244, 233], [238, 246, 235], [235, 246, 238], [234, 247, 238], [251, 243, 232], [250, 244, 232], [250, 244, 232], [248, 244, 232], [247, 245, 233], [244, 244, 232], [244, 244, 232], [243, 245, 234], [241, 244, 233], [241, 244, 233], [241, 244, 235], [241, 244, 235], [241, 244, 235], [241, 244, 237], [241, 244, 237], [241, 244, 237], [249, 229, 222], [255, 243, 236], [246, 227, 220], [255, 251, 243], [230, 217, 209], [147, 136, 130], [92, 83, 78], [154, 147, 141], [184, 179, 175], [219, 214, 211], [255, 254, 253], [237, 231, 233], [255, 253, 255], [255, 253, 255], [255, 252, 255], [255, 249, 254]], [[255, 252, 250], [255, 252, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [248, 251, 255], [251, 255, 255], [251, 255, 255], [247, 251, 254], [251, 255, 255], [254, 255, 255], [242, 243, 245], [250, 252, 251], [255, 252, 255], [241, 231, 239], [255, 251, 255], [255, 244, 247], [255, 251, 248], [171, 152, 146], [47, 28, 21], [201, 181, 172], [255, 238, 226], [255, 238, 228], [252, 239, 230], [248, 241, 233], [245, 242, 235], [242, 243, 238], [239, 244, 240], [238, 244, 242], [250, 240, 230], [249, 241, 230], [249, 241, 230], [248, 242, 230], [246, 242, 230], [245, 243, 231], [243, 243, 231], [242, 244, 231], [242, 244, 233], [242, 244, 233], [242, 244, 233], [242, 243, 235], [242, 243, 235], [243, 243, 235], [243, 243, 235], [243, 243, 235], [255, 244, 239], [255, 239, 233], [255, 244, 237], [206, 187, 181], [47, 32, 25], [143, 132, 126], [255, 253, 248], [255, 253, 249], [253, 249, 246], [255, 255, 253], [255, 254, 255], [250, 248, 249], [255, 254, 255], [247, 245, 250], [250, 248, 253], [255, 253, 255]], [[255, 252, 250], [255, 252, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [250, 255, 255], [248, 253, 255], [249, 253, 255], [250, 254, 255], [253, 254, 255], [252, 253, 255], [251, 253, 252], [248, 252, 255], [247, 250, 255], [253, 254, 255], [255, 253, 255], [255, 253, 253], [255, 252, 250], [205, 191, 188], [104, 89, 84], [100, 83, 76], [215, 198, 191], [242, 227, 222], [255, 246, 245], [255, 252, 251], [231, 222, 223], [255, 251, 255], [244, 238, 242], [253, 243, 233], [242, 232, 222], [244, 236, 225], [251, 245, 233], [243, 239, 227], [238, 236, 224], [244, 244, 232], [250, 250, 238], [247, 247, 235], [219, 219, 209], [255, 255, 246], [244, 241, 232], [247, 243, 234], [246, 242, 233], [249, 242, 234], [255, 252, 246], [255, 237, 232], [224, 199, 194], [118, 97, 92], [77, 58, 52], [219, 204, 199], [255, 252, 248], [252, 244, 241], [255, 254, 251], [255, 255, 253], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255]], [[255, 252, 250], [255, 253, 251], [255, 253, 251], [255, 253, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [251, 254, 255], [251, 254, 255], [252, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [244, 255, 255], [242, 255, 255], [248, 255, 255], [251, 255, 255], [242, 243, 245], [252, 250, 251], [255, 253, 253], [255, 248, 247], [214, 200, 199], [79, 63, 63], [118, 100, 100], [184, 165, 167], [210, 189, 194], [255, 249, 255], [244, 225, 231], [255, 239, 245], [255, 246, 238], [255, 243, 235], [254, 244, 235], [255, 247, 238], [255, 251, 241], [254, 250, 239], [244, 242, 230], [235, 233, 221], [250, 248, 236], [255, 253, 241], [255, 255, 244], [233, 226, 216], [255, 253, 244], [255, 252, 244], [219, 204, 197], [192, 177, 170], [122, 97, 93], [81, 57, 55], [188, 167, 164], [255, 250, 246], [255, 247, 244], [237, 226, 224], [255, 253, 250], [255, 252, 249], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255]], [[255, 253, 251], [255, 254, 251], [255, 253, 253], [255, 254, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [252, 252, 254], [250, 250, 252], [251, 249, 250], [255, 251, 250], [255, 254, 253], [255, 254, 251], [239, 255, 255], [239, 255, 255], [245, 255, 255], [248, 255, 255], [251, 255, 255], [252, 253, 255], [255, 254, 255], [255, 250, 252], [254, 242, 246], [245, 229, 232], [47, 26, 31], [127, 102, 106], [109, 80, 85], [103, 72, 78], [154, 121, 128], [205, 172, 179], [206, 191, 186], [227, 214, 208], [236, 225, 219], [234, 225, 216], [235, 228, 218], [234, 230, 219], [235, 231, 220], [242, 240, 228], [232, 228, 217], [215, 209, 197], [198, 190, 179], [179, 166, 157], [170, 153, 145], [97, 78, 71], [87, 64, 58], [166, 143, 137], [102, 81, 80], [218, 197, 196], [255, 249, 248], [254, 238, 238], [255, 251, 250], [255, 252, 251], [255, 254, 251], [250, 246, 245], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255]], [[255, 254, 251], [255, 254, 251], [255, 255, 253], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [251, 251, 253], [255, 254, 255], [255, 254, 255], [255, 253, 253], [255, 253, 251], [255, 252, 250], [255, 248, 246], [255, 246, 242], [249, 255, 253], [249, 255, 253], [241, 250, 247], [233, 239, 239], [240, 244, 245], [254, 255, 255], [255, 254, 255], [255, 248, 255], [255, 251, 255], [255, 248, 254], [132, 109, 115], [223, 192, 197], [255, 244, 248], [227, 187, 188], [208, 164, 165], [165, 119, 121], [117, 102, 99], [112, 98, 95], [97, 83, 80], [94, 85, 78], [111, 104, 96], [114, 107, 99], [101, 97, 86], [102, 95, 85], [102, 96, 84], [114, 104, 94], [123, 109, 98], [146, 128, 118], [187, 165, 154], [218, 191, 182], [249, 219, 211], [255, 233, 226], [143, 124, 126], [199, 180, 182], [255, 241, 242], [255, 251, 251], [255, 247, 246], [251, 242, 243], [255, 254, 253], [252, 250, 251], [255, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 254, 255], [255, 254, 255]], [[255, 255, 253], [255, 255, 253], [255, 255, 253], [255, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 249, 250], [254, 244, 242], [255, 241, 238], [255, 244, 239], [255, 250, 246], [255, 251, 244], [249, 242, 232], [255, 254, 244], [255, 254, 248], [255, 254, 251], [255, 255, 253], [255, 254, 255], [255, 251, 255], [249, 242, 249], [255, 251, 255], [249, 230, 236], [216, 191, 194], [73, 39, 40], [244, 203, 201], [255, 210, 204], [252, 199, 191], [255, 232, 222], [249, 233, 233], [237, 223, 222], [225, 211, 210], [223, 212, 210], [224, 215, 210], [216, 209, 201], [214, 207, 197], [223, 214, 205], [218, 208, 198], [249, 235, 224], [246, 228, 216], [255, 244, 232], [255, 247, 237], [255, 245, 235], [255, 242, 233], [255, 221, 213], [44, 29, 32], [239, 224, 227], [255, 251, 253], [236, 226, 227], [255, 252, 255], [255, 254, 255], [246, 244, 245], [255, 255, 255], [255, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255]], [[254, 255, 253], [254, 255, 253], [254, 255, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 253], [255, 253, 251], [255, 252, 250], [255, 251, 248], [255, 250, 246], [255, 244, 239], [255, 240, 234], [255, 239, 231], [255, 248, 233], [255, 249, 235], [255, 247, 236], [255, 251, 244], [255, 251, 250], [255, 251, 252], [255, 248, 253], [255, 251, 255], [250, 237, 244], [255, 249, 253], [255, 247, 248], [72, 37, 33], [93, 49, 40], [89, 37, 23], [162, 105, 86], [255, 239, 220], [255, 245, 248], [255, 240, 243], [255, 251, 251], [255, 252, 250], [252, 242, 240], [248, 239, 234], [255, 248, 241], [255, 249, 240], [255, 247, 235], [255, 246, 234], [243, 221, 208], [255, 232, 221], [252, 216, 204], [255, 238, 228], [255, 218, 207], [127, 80, 70], [155, 144, 150], [255, 251, 255], [255, 252, 255], [255, 253, 255], [255, 253, 255], [251, 249, 252], [249, 249, 251], [255, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255]], [[252, 255, 253], [252, 255, 253], [252, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [247, 241, 241], [255, 247, 246], [255, 251, 248], [255, 244, 240], [245, 225, 218], [223, 198, 191], [204, 177, 168], [194, 165, 157], [202, 161, 141], [212, 174, 155], [215, 182, 167], [234, 205, 197], [255, 239, 234], [255, 246, 247], [254, 242, 246], [255, 250, 255], [255, 240, 245], [255, 249, 250], [203, 175, 172], [126, 89, 80], [192, 146, 130], [136, 81, 60], [65, 5, 0], [226, 163, 132], [246, 229, 235], [242, 226, 229], [255, 244, 247], [255, 244, 244], [235, 225, 223], [252, 243, 238], [255, 253, 246], [232, 222, 213], [255, 252, 241], [224, 204, 193], [64, 37, 26], [132, 100, 87], [255, 230, 218], [231, 184, 174], [117, 67, 56], [69, 17, 6], [213, 206, 213], [255, 252, 255], [247, 240, 247], [255, 253, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [243, 244, 246], [254, 255, 255], [255, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255], [255, 253, 255], [255, 252, 255]], [[251, 255, 253], [251, 255, 255], [251, 255, 255], [251, 255, 255], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 254, 251], [229, 219, 217], [170, 155, 152], [113, 94, 88], [84, 61, 55], [91, 64, 55], [121, 93, 82], [147, 117, 107], [148, 98, 75], [156, 108, 88], [121, 78, 62], [86, 49, 40], [112, 84, 80], [171, 149, 151], [221, 205, 208], [255, 243, 248], [255, 244, 247], [255, 249, 250], [192, 164, 160], [143, 107, 93], [254, 207, 187], [229, 175, 147], [224, 164, 130], [112, 48, 12], [120, 103, 109], [186, 169, 175], [255, 246, 249], [232, 220, 222], [150, 140, 139], [173, 164, 159], [247, 238, 231], [255, 248, 240], [253, 236, 226], [163, 143, 132], [104, 76, 64], [139, 105, 93], [65, 23, 11], [99, 51, 39], [214, 162, 151], [174, 120, 108], [148, 141, 148], [255, 253, 255], [255, 252, 255], [250, 248, 253], [255, 254, 255], [255, 254, 255], [251, 252, 254], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255], [255, 252, 253], [255, 252, 253]], [[254, 255, 255], [252, 255, 255], [252, 255, 255], [252, 255, 255], [250, 250, 250], [255, 252, 246], [255, 245, 230], [232, 205, 178], [142, 103, 64], [107, 55, 5], [145, 85, 25], [227, 159, 94], [254, 184, 114], [251, 181, 109], [255, 192, 122], [255, 200, 129], [237, 205, 118], [245, 210, 126], [239, 203, 125], [239, 202, 131], [200, 160, 98], [153, 112, 58], [100, 56, 9], [121, 76, 35], [245, 200, 161], [255, 232, 193], [178, 133, 92], [184, 139, 97], [231, 187, 140], [239, 198, 146], [249, 208, 152], [230, 190, 131], [214, 154, 118], [92, 29, 0], [157, 93, 58], [147, 78, 45], [138, 69, 40], [124, 54, 29], [255, 201, 182], [255, 212, 195], [141, 84, 67], [128, 75, 57], [239, 187, 163], [246, 193, 159], [207, 153, 106], [252, 193, 135], [255, 198, 129], [218, 154, 80], [136, 79, 85], [255, 230, 235], [255, 242, 250], [255, 246, 253], [255, 244, 251], [255, 253, 255], [254, 255, 255], [249, 253, 252], [252, 255, 251], [255, 255, 250], [255, 255, 250], [255, 254, 251], [255, 253, 255], [255, 252, 255], [255, 252, 255], [255, 252, 255]], [[254, 252, 255], [246, 245, 250], [255, 254, 255], [241, 241, 243], [255, 254, 252], [242, 235, 227], [137, 121, 106], [89, 63, 36], [196, 158, 121], [215, 165, 116], [248, 189, 133], [255, 200, 138], [255, 191, 125], [255, 196, 128], [255, 203, 134], [253, 190, 123], [244, 201, 123], [240, 196, 121], [229, 185, 114], [248, 200, 138], [251, 202, 146], [242, 192, 143], [223, 170, 128], [217, 163, 125], [68, 14, 0], [115, 61, 25], [82, 28, 0], [206, 153, 111], [255, 215, 168], [235, 185, 134], [237, 188, 132], [236, 187, 129], [195, 137, 100], [242, 182, 145], [159, 97, 60], [228, 164, 128], [249, 183, 149], [149, 84, 54], [175, 111, 86], [140, 79, 58], [183, 128, 107], [222, 170, 148], [166, 116, 89], [217, 167, 130], [250, 198, 150], [246, 190, 131], [255, 194, 127], [235, 173, 100], [98, 40, 39], [105, 53, 55], [237, 196, 200], [255, 246, 251], [255, 250, 255], [255, 252, 255], [240, 241, 245], [252, 255, 255], [252, 255, 253], [255, 255, 250], [255, 255, 250], [255, 254, 251], [255, 254, 255], [255, 253, 255], [255, 253, 255], [255, 252, 255]], [[255, 246, 245], [255, 252, 253], [255, 253, 253], [253, 249, 248], [191, 186, 183], [24, 15, 8], [175, 158, 142], [244, 218, 193], [236, 200, 164], [242, 198, 153], [247, 196, 141], [249, 193, 134], [247, 188, 128], [243, 187, 126], [242, 188, 126], [239, 187, 127], [238, 180, 116], [244, 186, 123], [239, 180, 122], [246, 186, 134], [253, 190, 146], [254, 190, 152], [255, 206, 171], [228, 163, 131], [139, 74, 42], [194, 129, 97], [128, 64, 28], [237, 174, 133], [255, 196, 149], [236, 176, 124], [250, 191, 135], [226, 167, 109], [84, 29, 0], [228, 173, 132], [255, 197, 157], [250, 191, 149], [236, 174, 135], [182, 122, 85], [95, 36, 2], [207, 152, 121], [223, 172, 143], [255, 220, 190], [111, 65, 31], [198, 150, 110], [255, 215, 166], [238, 186, 126], [244, 187, 120], [245, 187, 114], [128, 68, 58], [189, 134, 127], [53, 9, 6], [230, 198, 199], [255, 247, 249], [255, 253, 255], [255, 255, 255], [253, 254, 255], [254, 255, 255], [255, 255, 251], [255, 255, 251], [255, 254, 251], [255, 254, 255], [255, 254, 255], [255, 254, 255], [255, 253, 255]], [[255, 247, 243], [255, 251, 248], [232, 218, 215], [127, 116, 114], [80, 69, 65], [234, 221, 212], [253, 236, 220], [254, 228, 203], [232, 198, 163], [229, 187, 145], [224, 178, 128], [234, 184, 131], [245, 196, 140], [239, 192, 136], [234, 189, 134], [242, 199, 146], [251, 184, 131], [255, 196, 144], [255, 194, 147], [255, 187, 145], [252, 180, 142], [241, 168, 136], [255, 182, 153], [165, 91, 64], [191, 117, 88], [250, 179, 147], [150, 80, 44], [238, 169, 127], [245, 177, 128], [255, 189, 136], [255, 216, 158], [225, 161, 99], [134, 81, 39], [234, 181, 137], [252, 198, 152], [243, 189, 142], [255, 210, 160], [247, 195, 147], [216, 165, 118], [254, 205, 162], [242, 197, 156], [235, 190, 151], [125, 80, 39], [203, 159, 114], [244, 197, 145], [242, 192, 133], [240, 186, 122], [244, 189, 122], [107, 45, 24], [255, 211, 194], [181, 133, 121], [141, 106, 100], [255, 249, 248], [243, 233, 234], [241, 239, 242], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 255, 253], [255, 254, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255]], [[255, 248, 241], [240, 217, 211], [58, 39, 33], [180, 162, 158], [240, 225, 218], [255, 245, 235], [255, 235, 218], [228, 202, 177], [217, 182, 150], [242, 203, 162], [249, 205, 158], [240, 194, 144], [238, 195, 142], [239, 198, 146], [232, 195, 143], [229, 193, 143], [255, 188, 139], [255, 188, 142], [252, 183, 141], [253, 184, 143], [255, 191, 155], [255, 185, 153], [248, 177, 147], [117, 46, 16], [235, 164, 134], [247, 179, 144], [127, 59, 20], [236, 169, 124], [250, 184, 132], [237, 172, 114], [255, 192, 130], [217, 153, 89], [133, 82, 39], [233, 182, 135], [243, 193, 142], [247, 198, 142], [238, 189, 130], [211, 165, 105], [255, 217, 158], [233, 190, 135], [247, 204, 153], [214, 170, 123], [145, 101, 54], [211, 165, 116], [240, 193, 141], [246, 197, 139], [249, 199, 138], [237, 186, 121], [131, 65, 31], [212, 151, 122], [255, 217, 194], [61, 23, 10], [255, 240, 235], [255, 251, 251], [255, 253, 255], [236, 235, 240], [255, 254, 255], [255, 255, 255], [255, 255, 255], [255, 255, 253], [255, 255, 255], [254, 255, 255], [251, 255, 255], [251, 255, 255]], [[236, 207, 199], [83, 56, 49], [192, 169, 163], [255, 238, 233], [254, 235, 228], [255, 245, 234], [255, 245, 227], [255, 228, 201], [255, 224, 189], [246, 204, 162], [236, 192, 143], [239, 193, 141], [235, 192, 139], [229, 188, 136], [226, 189, 137], [230, 194, 144], [243, 182, 128], [245, 183, 132], [245, 183, 134], [246, 185, 140], [246, 184, 143], [251, 189, 152], [228, 168, 132], [112, 52, 16], [252, 192, 156], [240, 181, 141], [117, 58, 14], [222, 164, 114], [255, 205, 150], [248, 192, 131], [255, 211, 147], [249, 194, 129], [136, 81, 40], [207, 155, 108], [255, 205, 152], [237, 191, 131], [230, 185, 118], [255, 220, 151], [226, 186, 116], [240, 199, 133], [245, 200, 141], [232, 187, 132], [130, 82, 33], [207, 157, 108], [255, 205, 156], [242, 191, 138], [253, 202, 147], [228, 176, 119], [137, 66, 22], [229, 166, 125], [233, 180, 148], [98, 57, 37], [214, 187, 178], [255, 244, 244], [255, 253, 255], [254, 251, 255], [255, 254, 255], [255, 254, 255], [255, 255, 255], [255, 255, 255], [254, 255, 253], [251, 255, 253], [249, 255, 255], [248, 255, 255]], [[66, 36, 28], [166, 137, 131], [246, 221, 216], [255, 249, 243], [240, 220, 213], [242, 220, 207], [255, 246, 226], [255, 248, 220], [255, 248, 211], [239, 195, 150], [228, 178, 127], [247, 196, 141], [246, 197, 139], [238, 191, 135], [241, 198, 143], [246, 205, 149], [244, 195, 129], [249, 199, 136], [253, 203, 144], [251, 202, 146], [233, 183, 132], [244, 196, 148], [206, 159, 113], [126, 79, 35], [236, 189, 145], [240, 193, 147], [136, 90, 41], [203, 157, 105], [253, 206, 150], [236, 190, 130], [232, 186, 124], [209, 164, 99], [128, 69, 29], [227, 170, 125], [248, 195, 143], [240, 192, 130], [230, 188, 116], [244, 202, 126], [217, 175, 99], [243, 201, 129], [246, 198, 134], [233, 182, 125], [118, 65, 15], [223, 167, 120], [253, 196, 151], [243, 187, 140], [248, 195, 145], [224, 171, 121], [138, 67, 13], [231, 165, 117], [255, 204, 167], [115, 73, 48], [232, 202, 191], [255, 250, 248], [245, 236, 241], [255, 253, 255], [255, 254, 255], [255, 254, 255], [255, 254, 255], [254, 255, 255], [252, 255, 253], [251, 255, 253], [248, 255, 253], [247, 255, 253]], [[223, 195, 191], [76, 51, 46], [197, 173, 169], [255, 249, 244], [255, 250, 243], [255, 242, 229], [255, 236, 218], [248, 213, 185], [235, 191, 152], [236, 186, 137], [255, 201, 145], [255, 199, 139], [238, 180, 117], [246, 191, 127], [254, 203, 140], [230, 180, 117], [245, 202, 124], [234, 192, 116], [231, 189, 115], [238, 198, 129], [231, 192, 127], [255, 216, 157], [193, 154, 99], [120, 80, 29], [238, 198, 149], [233, 193, 144], [135, 95, 46], [187, 145, 95], [247, 206, 154], [230, 187, 134], [237, 194, 139], [247, 202, 147], [138, 72, 37], [220, 157, 116], [253, 195, 145], [238, 188, 127], [245, 201, 130], [243, 201, 125], [233, 190, 112], [248, 201, 129], [254, 203, 138], [213, 154, 98], [134, 72, 25], [255, 194, 152], [238, 175, 134], [255, 194, 155], [247, 188, 146], [231, 174, 131], [137, 65, 6], [234, 169, 115], [255, 211, 170], [117, 73, 44], [216, 187, 173], [255, 250, 246], [255, 252, 255], [255, 253, 255], [255, 253, 255], [255, 254, 255], [254, 255, 255], [254, 255, 255], [252, 255, 253], [249, 255, 251], [248, 255, 251], [247, 255, 251]], [[255, 248, 246], [220, 199, 198], [45, 25, 24], [174, 156, 154], [255, 237, 231], [255, 245, 234], [247, 219, 198], [231, 194, 165], [237, 189, 149], [246, 190, 139], [252, 190, 131], [255, 190, 124], [255, 189, 121], [253, 189, 118], [252, 189, 120], [251, 191, 121], [238, 194, 107], [253, 208, 123], [238, 196, 114], [228, 187, 108], [240, 200, 128], [248, 207, 143], [202, 162, 103], [118, 77, 23], [233, 191, 141], [241, 198, 153], [147, 102, 60], [180, 133, 91], [250, 201, 160], [228, 176, 136], [246, 193, 151], [252, 199, 157], [180, 105, 76], [189, 119, 85], [255, 197, 153], [245, 193, 135], [248, 201, 133], [240, 196, 123], [232, 186, 111], [248, 198, 129], [250, 191, 131], [211, 145, 95], [149, 76, 35], [255, 201, 166], [248, 175, 143], [252, 183, 150], [255, 194, 159], [200, 137, 102], [143, 73, 13], [255, 204, 150], [244, 191, 149], [98, 57, 27], [224, 196, 182], [255, 250, 246], [255, 248, 253], [255, 253, 255], [255, 253, 255], [254, 254, 255], [254, 255, 255], [254, 255, 255], [252, 255, 253], [249, 255, 251], [248, 255, 251], [247, 255, 251]], [[255, 239, 240], [255, 250, 251], [241, 227, 227], [101, 87, 87], [117, 102, 99], [238, 220, 210], [255, 249, 230], [234, 197, 168], [240, 193, 151], [241, 184, 131], [243, 177, 117], [250, 181, 114], [255, 190, 120], [255, 195, 123], [255, 192, 121], [253, 185, 114], [245, 191, 103], [255, 204, 118], [238, 186, 103], [248, 196, 120], [246, 196, 125], [249, 199, 136], [226, 175, 118], [105, 53, 5], [221, 168, 124], [250, 194, 157], [212, 153, 121], [97, 33, 5], [233, 168, 140], [227, 160, 134], [198, 128, 103], [236, 164, 140], [193, 111, 90], [177, 101, 75], [237, 172, 134], [247, 191, 140], [241, 193, 131], [240, 195, 128], [248, 200, 134], [236, 182, 122], [255, 195, 145], [179, 104, 64], [201, 120, 90], [239, 158, 131], [251, 171, 148], [241, 167, 142], [255, 193, 166], [174, 109, 81], [185, 118, 63], [244, 184, 134], [240, 191, 151], [93, 56, 29], [253, 227, 214], [255, 251, 248], [238, 232, 236], [254, 253, 255], [254, 254, 255], [254, 255, 255], [254, 255, 255], [254, 255, 255], [252, 255, 253], [251, 255, 253], [249, 255, 253], [248, 255, 253]], [[255, 251, 255], [250, 239, 243], [254, 245, 248], [255, 249, 252], [164, 154, 153], [36, 21, 14], [203, 180, 164], [238, 204, 177], [244, 199, 160], [255, 200, 151], [255, 198, 142], [255, 191, 129], [254, 183, 117], [250, 180, 111], [253, 184, 117], [255, 190, 122], [252, 189, 110], [255, 202, 125], [245, 183, 110], [255, 197, 127], [235, 173, 112], [239, 178, 123], [255, 201, 154], [154, 91, 50], [212, 146, 112], [252, 182, 156], [171, 96, 75], [100, 21, 6], [97, 13, 2], [104, 17, 8], [136, 46, 38], [130, 39, 34], [132, 46, 33], [120, 42, 22], [244, 178, 146], [255, 209, 165], [224, 178, 126], [229, 187, 129], [235, 188, 132], [247, 191, 142], [251, 183, 144], [145, 66, 36], [122, 35, 15], [171, 84, 67], [184, 101, 85], [185, 108, 90], [230, 162, 141], [128, 64, 39], [209, 147, 100], [255, 214, 170], [191, 145, 111], [122, 90, 67], [253, 233, 222], [252, 242, 240], [255, 254, 255], [253, 254, 255], [254, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 255], [254, 255, 253], [252, 255, 253], [251, 255, 255], [249, 255, 255]], [[255, 253, 255], [255, 250, 254], [255, 253, 255], [248, 243, 247], [255, 253, 255], [226, 217, 212], [95, 77, 65], [81, 53, 32], [187, 148, 117], [216, 167, 126], [248, 192, 143], [255, 203, 148], [255, 199, 142], [254, 192, 135], [250, 189, 132], [251, 190, 135], [252, 184, 121], [252, 184, 123], [237, 171, 113], [255, 196, 142], [255, 196, 148], [255, 193, 151], [252, 186, 151], [167, 100, 71], [132, 60, 38], [209, 133, 119], [77, 0, 0], [118, 30, 26], [118, 24, 24], [130, 31, 34], [147, 46, 52], [126, 22, 29], [124, 39, 32], [84, 6, 0], [219, 153, 131], [246, 193, 159], [239, 196, 153], [255, 229, 183], [227, 182, 140], [253, 197, 162], [231, 161, 135], [86, 4, 0], [126, 37, 29], [115, 24, 19], [91, 6, 0], [128, 52, 39], [101, 34, 17], [128, 67, 46], [202, 146, 109], [204, 154, 121], [151, 111, 85], [202, 175, 158], [255, 247, 239], [255, 254, 251], [250, 251, 253], [243, 247, 250], [252, 255, 255], [254, 255, 255], [255, 255, 255], [255, 255, 253], [255, 255, 255], [255, 255, 255], [254, 255, 255], [252, 255, 255]], [[254, 250, 251], [249, 247, 250], [244, 243, 248], [253, 252, 255], [235, 235, 237], [255, 254, 253], [255, 244, 238], [219, 200, 185], [123, 93, 69], [138, 99, 68], [163, 118, 79], [200, 151, 110], [234, 185, 142], [253, 204, 163], [253, 205, 165], [244, 199, 158], [252, 191, 147], [255, 200, 158], [255, 204, 164], [248, 190, 153], [222, 166, 133], [182, 125, 96], [142, 84, 62], [141, 80, 62], [101, 37, 25], [86, 15, 9], [114, 39, 36], [113, 31, 33], [130, 41, 47], [131, 36, 44], [126, 26, 36], [114, 12, 25], [128, 44, 44], [96, 21, 15], [149, 87, 72], [187, 140, 114], [200, 164, 130], [214, 180, 143], [194, 155, 122], [229, 177, 155], [173, 107, 93], [85, 4, 1], [137, 48, 50], [107, 18, 22], [125, 44, 41], [117, 44, 37], [89, 27, 12], [128, 73, 53], [114, 64, 39], [81, 36, 15], [59, 26, 9], [237, 215, 204], [255, 252, 248], [239, 238, 236], [251, 255, 255], [249, 255, 255], [251, 255, 255], [254, 255, 255], [255, 255, 253], [255, 255, 253], [255, 254, 255], [255, 254, 255], [255, 253, 255], [255, 254, 255]], [[255, 254, 253], [255, 254, 255], [254, 255, 255], [252, 255, 255], [252, 255, 255], [254, 255, 255], [255, 254, 251], [255, 245, 236], [254, 235, 220], [217, 190, 169], [159, 127, 102], [115, 81, 53], [103, 69, 42], [117, 85, 60], [143, 113, 89], [160, 132, 110], [167, 125, 101], [159, 117, 93], [146, 105, 83], [110, 69, 49], [118, 80, 61], [147, 108, 91], [186, 146, 134], [255, 242, 233], [220, 173, 167], [66, 13, 9], [129, 69, 69], [124, 58, 60], [103, 28, 33], [103, 22, 29], [131, 45, 54], [143, 56, 65], [108, 29, 34], [116, 46, 44], [87, 30, 19], [154, 113, 93], [147, 117, 91], [96, 70, 43], [127, 95, 74], [140, 94, 81], [103, 39, 37], [135, 58, 64], [100, 14, 25], [130, 44, 53], [134, 55, 58], [105, 38, 32], [115, 62, 46], [99, 52, 32], [111, 68, 52], [71, 33, 20], [210, 181, 173], [255, 241, 237], [253, 248, 245], [252, 255, 255], [233, 242, 241], [248, 255, 255], [249, 255, 255], [252, 255, 251], [255, 255, 251], [255, 254, 251], [255, 253, 255], [255, 252, 255], [255, 252, 255], [255, 252, 255]], [[255, 254, 251], [237, 236, 234], [254, 255, 255], [251, 255, 255], [234, 241, 247], [251, 255, 255], [254, 255, 255], [248, 244, 241], [255, 250, 241], [255, 252, 239], [255, 250, 236], [255, 237, 223], [227, 208, 194], [196, 178, 166], [174, 160, 151], [165, 152, 144], [173, 150, 142], [168, 148, 139], [179, 159, 150], [204, 186, 176], [253, 234, 227], [255, 250, 243], [255, 250, 243], [255, 249, 243], [255, 247, 243], [180, 146, 144], [61, 21, 21], [70, 22, 22], [88, 33, 36], [102, 42, 44], [73, 7, 11], [76, 7, 12], [104, 29, 36], [97, 33, 34], [89, 39, 32], [224, 191, 176], [255, 237, 216], [242, 221, 202], [255, 247, 233], [254, 213, 209], [71, 12, 18], [82, 8, 23], [105, 22, 40], [109, 27, 41], [68, 0, 2], [110, 49, 44], [51, 4, 0], [78, 37, 15], [66, 25, 19], [145, 110, 106], [255, 237, 236], [255, 251, 251], [255, 253, 254], [251, 255, 255], [247, 255, 255], [234, 246, 242], [249, 255, 253], [252, 255, 250], [255, 255, 250], [255, 254, 251], [255, 252, 255], [255, 251, 255], [255, 251, 255], [255, 250, 255]], [[240, 237, 232], [255, 255, 251], [238, 240, 239], [248, 253, 255], [249, 255, 255], [249, 255, 255], [240, 245, 249], [255, 255, 255], [255, 250, 246], [255, 248, 242], [255, 248, 239], [255, 250, 241], [255, 252, 246], [255, 252, 250], [255, 253, 253], [254, 252, 253], [255, 247, 245], [255, 253, 251], [255, 254, 251], [255, 254, 251], [250, 246, 243], [250, 246, 243], [255, 254, 250], [248, 240, 237], [255, 242, 239], [255, 250, 246], [215, 190, 186], [195, 161, 159], [187, 146, 144], [255, 230, 230], [255, 207, 207], [176, 120, 121], [139, 68, 76], [104, 42, 45], [163, 115, 111], [255, 246, 233], [255, 250, 233], [255, 252, 236], [255, 249, 239], [255, 238, 237], [190, 134, 145], [103, 30, 49], [153, 71, 93], [177, 96, 113], [251, 180, 188], [255, 236, 232], [222, 179, 163], [179, 143, 121], [230, 192, 191], [255, 246, 244], [255, 246, 248], [255, 251, 253], [244, 244, 246], [245, 253, 255], [243, 255, 255], [247, 255, 255], [249, 255, 251], [252, 255, 250], [255, 255, 250], [255, 254, 251], [255, 252, 255], [255, 251, 255], [255, 250, 255], [255, 250, 255]]]]
nd_test = np.asarray(test)

inspect = tf.contrib.util.make_tensor_proto(nd_test, dtype='uint8', shape=nd_test.shape)

request.inputs['inputs'].CopyFrom(inspect)

start = int(round(time.time() * 1000))
tt, call = stub.Predict(request, 3, with_call=True)

With this script even for ONE SINGLE request, as you can see i set the predict timeout to 3s stub.Predict(request, 3, with_call=True), the client got the error:

Traceback (most recent call last):
  File "python2_insecure.py", line 37, in <module>
    tt, call = stub.Predict(request, 3, with_call=True)
  File "/usr/local/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 309, in __call__
    self._request_serializer, self._response_deserializer)
  File "/usr/local/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 195, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.ExpirationError: ExpirationError(code=StatusCode.DEADLINE_EXCEEDED, details="Deadline Exceeded")

But on TFS side, it shows (we just added logs to print the request duration):

2019-05-13 04:59:32.022336: I tensorflow_serving/model_servers/prediction_service_impl.cc:41] Clea Request Summary  ModelName: [inception] Success: [0] RequestDurationMillis: [6181]

So the server insists to fulfil the request by consuming more than 6s, though the request deadline is set to 3s.

This is just one single request case, for you to easily inspect the problem.

@lnshi I know this is out of context here. But I really wanted to know how to add the request logs on TFS side. I did an extensive research on this and failed miserably. Hope you can help me with this. Thanks.

sathuhareesh on 31 Oct 2019

@sathuhareesh modify the source code, and compile it.

lnshi on 31 Oct 2019

@sathuhareesh modify the source code, and compile it.

We are actually using tensorflow/serving docker image on server side. So, is there any way I can implement this on docker?

sathuhareesh on 31 Oct 2019

for custom logging i don't think so

lnshi on 31 Oct 2019

for custom logging i don't think so

Okay! Anyways thank you very much for the response.

sathuhareesh on 31 Oct 2019

channel

Hi @lnshi do you know how the grpc channel works? Like, currently for each request I'm closing the channel. Is this recommended to close it or keep it open like that?

sathuhareesh on 1 Nov 2019

Is there any update for this issue? Running into the same issue here. Thanks

gyin-ai on 25 Nov 2020

This issue is due to proxy issue. To get rid of Deadline Exceed issue, try:

import os
if os.environ.get("http_proxy"):
    del os.environ["http_proxy"]
if os.environ.get("https_proxy"):
    del os.environ["https_proxy"]

apoorvmishra28 on 11 Jan 2021

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Export and run inference on modified InceptionV3 trained model

sskgit · 4Comments

[Question] How do I use tensorflow_serving deployed on kubernetes?

atwj · 4Comments

Load-balancing for real-time production environments in TensorFlow

johnsrude · 4Comments

Installation with GPU support ?

daikankan · 4Comments

Deploying a single model with high # of requests.

farzaa · 3Comments