Loopback: POST http://localhost:3000/api/rawSensorData 413 (Request Entity Too Large)

[kyuwoo-choi]

requesting post with long text data gives me '413 Request Entity Too Large'
after little digging, I found some answers from stackoverflow about express body parser.

I've tried below code in server.js but I have no luck.

app.use(loopback.bodyParser.json({limit: '1024mb'}));
app.use(loopback.bodyParser.urlencoded({limit: '1024mb', extended: true}));

how to solve this issue?

[raymondfeng]

For json/urlencoded limit, it’s recommended to configure them in server/config.json as follows:

{
“remoting”: {
“json”: {“limit”: “50mb”},
“urlencoded”: {“limit”: “50mb”, “extended”: true}
}
}

Please note loopback REST api has its own express router with bodyParser.json/urlencoded middleware. When you add a global middleware, it has to come before the boot() call.

Thanks,

---

Raymond Feng
Co-Founder and Architect @ StrongLoop, Inc.

StrongLoop http://strongloop.com/ makes it easy to develop APIs http://strongloop.com/mobile-application-development/loopback/ in Node, plus get DevOps capabilities http://strongloop.com/node-js-performance/strongops/ like monitoring, debugging and clustering.

On Oct 23, 2014, at 8:24 AM, KyuWoo Choi [email protected] wrote:

requesting post with long text data gives me '413 Request Entity Too Large'
after little digging, I found some answers from stackoverflow about express body parser.

I've tried below code in server.js but I have no luck.

app.use(loopback.bodyParser.json({limit: '1024mb'}));
app.use(loopback.bodyParser.urlencoded({limit: '1024mb', extended: true}));

how to solve this issue?

—
Reply to this email directly or view it on GitHub https://github.com/strongloop/loopback/issues/690.

[kyuwoo-choi]

thanks for quick answer :)
btw. I solved by moving previous middleware statements before all the other middleware statements like

var loopback = require('loopback');
var boot = require('loopback-boot');

var app = module.exports = loopback();

//request limit 1gb
app.use(loopback.bodyParser.json({limit: 524288000}));
app.use(loopback.bodyParser.urlencoded({limit: 524288000, extended: true}));

I think boot() call replaced middleware like you mentioned.
I'll try removing my solution and your config.

[gartox]

I have the same error for an endpoint expecting an application/json

Request Entity too large. I have this config on the middleware.json, but not works.

"parse:before": {
"body-parser#json": {
"limit": "500mb"
},
"body-parser#urlencoded": {"params": {"limit":"500mb", "extended": true }}
},

[alexxgathp]

The config.json solution did not work for me, the size stayed the same for some reason, but the solution above by @kyuwoo-choi worked.

Also, the size limit of the request was being hit by a PUT for me. It was directly related to a relation I was pulling in with an "include" filter. Because the child entities were so large, after five of them, the PUT 413 error started to occur.
If I am correct about the child entities being involved in the PUT done by $save(), I am not sure if that is the intent of the API's design, to have an update request that grows with the child entities, however, it may be.

In my situation, ultimately, I will need to re-design my application to query the child entities separately when I need them, since as-is they will grow the PUT request too fast still.

[raymondfeng]

You should not set up the json parser as there is already one configured per model. Adding one before that will shadow the per model ones which are configurable via server/config.json

[alexxgathp]

I just discovered what was causing interference with my config.json. I commented out my custom server.js stuff one-by-one to figure it out.

The lines interfering with config.json were definitions for a dataSource. I am guessing they should be elsewhere, though the documentation is a bit spotty about where it goes, I will try to find out, maybe a boot script?

var ds = loopback.createDataSource({
    connector: require('loopback-component-storage'),
    provider: 'filesystem',
    root: '/var/automate_files'
});

var container = ds.createModel('container');

config.json

{
    "restApiRoot": "/api",
    "host": "localhost",
    "port": 3000,
    "url": "http://localhost:3000/",
    "remoting": {
        "json": { "limit": "50mb" }, 
        "urlencoded": {"limit": "50mb", "extended": true},
        "cors": {
            "origin": true,
            "credentials": true
        }
    }
}

Error info

{"error":{"name":"Error","status":413,"message":"request entity too large","type":"entity.too.large"
,"statusCode":413,"expected":160670,"length":160670,"limit":102400,"stack":"Error: request entity too
 large\n    at makeError (/var/dev_automate/node_modules/loopback/node_modules/body-parser/node_modules
/raw-body/index.js:184:15)\n    at module.exports (/var/dev_automate/node_modules/loopback/node_modules
/body-parser/node_modules/raw-body/index.js:40:15)\n    at read (/var/dev_automate/node_modules/loopback
/node_modules/body-parser/lib/read.js:64:3)\n    at jsonParser (/var/dev_automate/node_modules/loopback
/node_modules/body-parser/lib/types/json.js:116:5)\n    at Layer.handle [as handle_request] (/var/dev_automate
/node_modules/loopback/node_modules/express/lib/router/layer.js:82:5)\n    at trim_prefix (/var/dev_automate
/node_modules/loopback/node_modules/express/lib/router/index.js:302:13)\n    at /var/dev_automate/node_modules
/loopback/node_modules/express/lib/router/index.js:270:7\n    at Function.proto.process_params (/var
/dev_automate/node_modules/loopback/node_modules/express/lib/router/index.js:321:12)\n    at next (/var
/dev_automate/node_modules/loopback/node_modules/express/lib/router/index.js:261:10)\n    at urlencodedParser
 (/var/dev_automate/node_modules/loopback/node_modules/body-parser/lib/types/urlencoded.js:91:37)"}}

[alexxgathp]

Okay, I think I found an example. It was buried down the page.
I'll try defining the datasource like the model on the boot script.
http://docs.strongloop.com/display/public/LB/Connect+your+API+to+a+data+source
http://docs.strongloop.com/display/public/LB/Defining+boot+scripts

[sjmcdowall]

OK -- I have been knocking my head on the wall about an issue like this for a day, and it's getting frustrating. I am setting up a new Loopback app, that does 2 things -- has a nice REST Api for a data model (normal stuff) AND also accepts POST commands (but NOT model related RESTFul posts) that takes data packets and basically inserts data into the data store (long story as to why they just aren't normal POST commands to the data model).

So I am getting the dreaded Entity too large when some of the POST commands hit. I have tried increasing the JSON/URLENCODE limits in MULTIPLE places, but nothing seems to "take".

I have the following snippet in the middleware.json :

  "session": {},
  "auth": {},
  "parse": {
    "body-parser#json": { "limit" : "50mb"},
    "body-parser#urlencoded": {"params": { "limit" : "50mb", "extended": true }}
  },
  "routes": {
    "loopback#rest": {
      "paths": [
        "${restApiRoot}"
      ]
    }
  },

I have added the following to the server.js file :
`
// Increase the body-parser middleware limit sizes
app.use(loopback.bodyParser.json({limit: 50 * 1024 * 1024}));
app.use(loopback.bodyParser.urlencoded({limit: '50mb', extended: true}));

// Bootstrap the application, configure models, datasources and middleware.
// Sub-apps like REST API are mounted via boot scripts.
boot(app, __dirname, function(err) {
if (err) throw err;`

... I have added the following to the config.json file:

    "rest": {
      "normalizeHttpPath": false,
      "xml": false
    },
    "json": {
      "strict": false,
      "limit": "50mb"
    },
    "urlencoded": {
      "extended": true,
      "limit": "50mb"
    },

But I STILL get the Entity too large error. Grrr..

My routes are defined in a boot/tems.js file like so:

`var fs = require('fs') ;
var util = require('util');

console.log('Setting up TEMS capture for CZ Data...') ;

module.exports = function(app) {
// Install the various TEMS calls we are interested in ..

var API_HEADER = '/TEMS/TEST_CELL/:cellId/' ; // Common URL header
var API_LOT_HEADER = API_HEADER + 'LOT/:lotId/' ;
var API_SUBLOT_HEADER = API_LOT_HEADER + 'SUBLOT/:sublotId/' ;

var replayStream ; // SHould be a hash of these when we do more than 1 TESTER...

// INITIALIZE -- first message to be called (we hope)
// Here we send back a list of messages we want ..
app.post( API_HEADER + 'INITIALIZATION', function(req, res) {

console.log('TEMS - INITIALIZATION RECEIVED ', req.body) ;

var wantedMessages = '["STATUS", "CONFIGURATION" , ' +
    '"TESTER_OS", "TEST_PROGRAM_LOAD" , "LOT_START", "MAINTENANCE",' +
    '"LOT_END", "SUBLOT_START", "SUBLOT_END",' +
    '"CZ_START", "CZ_SETUP", "CZ_START_DATA",' +
    '"CZ_DATA", "CZ_SUMMARY", "CZ_END",' +
    '"TEST_PROGRAM_LOAD", "SHUTDOWN"]' ;

res.status(200).json( wantedMessages );

// Open up replay stream to save the data packets ..
replayStream = fs.createWriteStream( req.params.cellId + '-replay.txt') ;

replayStream.write( JSON.stringify( req.body ) ) ;

});

// STATUS -- Heartbeat message ...
app.post( API_HEADER + 'STATUS', function(req, res) {

console.log('TEMS - STATUS RECEIVED', req.body) ;

res.status(200).end();

});

// CONFIGURATION -- How the tester is configured..
app.post( API_HEADER + 'CONFIGURATION', function(req, res) {

console.log('TEMS - CONFIGURATION RECEIVED', req.body) ;

res.status(200).end();

});

...

`

This should be pretty basic and simple I thought. Any/all help appreciated!

[alexxgathp]

My guess - if the config.json is not working to fix it - either the syntax or setting there is incorrect for it, or for a file loaded with it, or some other setting or line in the other code is interfering and resetting the value.

Also, if you are using an earlier version of loopback, it was buggy with the datasource settings, and for some reason would cause me to be unable to define the datasource in the app.js / server.js (resulting in screwing up my config.json settings). That is probably not your issue, but something to consider.

This is my config.json on the system that had the "Entity too large" issue.
{ "restApiRoot": "/api", "host": "localhost", "port": 7777, "url": "http://localhost:7777/", "remoting": { "json": { "limit": "50mb" }, "urlencoded": {"limit": "50mb", "extended": true}, "cors": { "origin": true, "credentials": true } } }

I am using one of the latest versions of loopback, and I do have some custom code too, but just a boot script creating data. I moved defining datasources back into the app.js / server.js (different now from my posts above). The earlier version did not load the datasources correctly with it there for some reason.

I would go through and see how your installation differs from a fresh installation of loopback with the modified config.json and that might give a clue what is causing it.

Also, one other detail, I don't have a middleware.json. I never got around to adding it when I upgraded to the latest version, so I don't know how it would affect all that.

[sjmcdowall]

I just downloaded / installed strong loop etc. so it's as new as it gets .. and generated a new project with it, which is how I got the middleware.json in the first place. So the scaffolding should be pretty spot on.

Middleware.json for me was pretty important because before I added anything there -- I was not even seeing a .body in the request for the app.post() routes I defined. Although I guess I can't be POSITIVE that was the change, but it was the one that I think caused some different.

I am not sure if the config.json is applicable to NON /api/ type calls -- there is some confusing documentation (and above explanations) about "model" RESTFUL api calls and non JSON calls. These are JSON calls with POST but are not related to the /api/ models I have.

My data source definition is pure in datasources.json (also generated by the framework) and that seems to work ok I think (not doing much with it now, but it says it auto created the CZJob model, the only thing I have defined so far).

And I am 99% sure my config.json is fine.. here is the whole file -- no complains about anything, etc.
{ "restApiRoot": "/api", "host": "0.0.0.0", "port": 3000, "remoting": { "context": { "enableHttpContext": false }, "rest": { "normalizeHttpPath": false, "xml": false }, "json": { "strict": false, "limit": "50mb" }, "urlencoded": { "extended": true, "limit": "50mb" }, "cors": false, "errorHandler": { "disableStackTrace": false } }, "legacyExplorer": false }

Here is a code snippet from the boot file that has the app.post middleware stuff :
`var fs = require('fs') ;
var util = require('util');

console.log('Setting up TEMS capture for CZ Data...') ;

module.exports = function(app) {
// Install the various TEMS calls we are interested in ..

var API_HEADER = '/TEMS/TEST_CELL/:cellId/' ; // Common URL header
var API_LOT_HEADER = API_HEADER + 'LOT/:lotId/' ;
var API_SUBLOT_HEADER = API_LOT_HEADER + 'SUBLOT/:sublotId/' ;

var replayStream ; // SHould be a hash of these when we do more than 1 TESTER...

// INITIALIZE -- first message to be called (we hope)
// Here we send back a list of messages we want ..
app.post( API_HEADER + 'INITIALIZATION', function(req, res) {

console.log('TEMS - INITIALIZATION RECEIVED ', req.body) ;

var wantedMessages = '["STATUS", "CONFIGURATION" , ' +
    '"TESTER_OS", "TEST_PROGRAM_LOAD" , "LOT_START", "MAINTENANCE",' +
    '"LOT_END", "SUBLOT_START", "SUBLOT_END",' +
    '"CZ_START", "CZ_SETUP", "CZ_START_DATA",' +
    '"CZ_DATA", "CZ_SUMMARY", "CZ_END",' +
    '"TEST_PROGRAM_LOAD", "SHUTDOWN"]' ;

res.status(200).json( wantedMessages );

// Open up replay stream to save the data packets ..
replayStream = fs.createWriteStream( req.params.cellId + '-replay.txt') ;

replayStream.write( JSON.stringify( req.body ) ) ;

});

// STATUS -- Heartbeat message ...
app.post( API_HEADER + 'STATUS', function(req, res) {

console.log('TEMS - STATUS RECEIVED', req.body) ;

res.status(200).end();

});

// CONFIGURATION -- How the tester is configured..
app.post( API_HEADER + 'CONFIGURATION', function(req, res) {

console.log('TEMS - CONFIGURATION RECEIVED', req.body) ;

res.status(200).end();

});
`

Cheers

[sjmcdowall]

And -- SOLVED -- it wasn't any issue in config.json it was middleware.json. Basically did not have the right "syntax" to properly overwrite the json limit parameter. The PROPER middleware.json looks something like:
```{
"initial:before": {
"loopback#favicon": {}
},
"initial": {
"compression": {},
"cors": {
"params": {
"origin": true,
"credentials": true,
"maxAge": 86400
}
}
},
"session": {},
"auth": {},
"parse": {
"body-parser#json": { "params" : { "limit" : "50mb"} },
"body-parser#urlencoded": {"params": { "limit" : "50mb", "extended": true }}
},
"routes": {
"loopback#rest": {
"paths": [
"${restApiRoot}"
]
}
},
"files": {
"loopback#static": {
"params": "$!../client"
}
},
"final": {
"loopback#urlNotFound": {}
},
"final:after": {
"loopback#errorHandler": {}
}
}

Note the "params" sub-element for JSON and URLEndcoding.  This is needed to properly set those. 

Phew!

[barocsi]

Can someone clarify this?
Its not clear in the docs how to do this properly. Do we even need to overwrite it in the middleware to have JSON limit increased? For me it throws 413 when trying to upload a large payload in JSON.

[shide1989]

thank you @sjmcdowall ! this saved me a couple hours of deep searching github...

[demsey2]

can anyone clarify what steps and config is required to get this working

[demsey2]

in case someone else has that problem and is running server on AWS EB here is the link to some additional settings
http://stackoverflow.com/questions/18908426/increasing-client-max-body-size-in-nginx-conf-on-aws-elastic-beanstalk

[itsvicsoto]

@sjmcdowall solution works, i tried paying attention to the middleware.json and yes it was missing sub-element params works like magic. Before that, I already tried adding bodyParser middlewares on my app.js and adding it before the bootscript, checked the config.json finally it's the middleware.json that's not correct.

[hans118]

Hi ,
I have build my own loopback app.
It is working in my local pc.
So I have delivered that into nginx server in linux 14.04.
But it is not working properly now.
For example, I can see these errors:
vendor.3aa8a0fe.js:41 Error: [$resource:badcfg] Error in resource configuration for action find. Expected response to contain an array but got an object
I can't understand why these errors is generating in nginx server.
In nginx, I am using upstream proxy for api root.
Please help me
:( :(

[hans118]

^-^

[sjmcdowall]

Sorry I can’t be of much help — I haven’t used Strongloop in over 2+ years and I actually have never used NGINX at all ..

Currently I’m doing pure Express work with FeathersJS

On May 10, 2017, at 1:28 PM, Hans Gradl [e-mail] wrote:

Hi ,
I have build my own loopback app.
It is working in my local pc.
So I have delivered that into nginx server in linux 14.04.
But it is not working properly now.
For example, I can see these errors:
vendor.3aa8a0fe.js:41 Error: [$resource:badcfg] Error in resource configuration for action find. Expected response to contain an array but got an object
I can't understand why these errors is generating in nginx server.
In nginx, I am using upstream proxy for api root.
Please help me
:( :(

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/strongloop/loopback/issues/690#issuecomment-300554717, or mute the thread https://github.com/notifications/unsubscribe-auth/AB8M7UKGiViCCPW6_aKPoOVW5IdD2XUQks5r4fPDgaJpZM4CyOPB.

[hans118]

Anyway, thanks for your reply.
But how do you think about this error?
vendor.3aa8a0fe.js:41 Error: [$resource:badcfg] Error in resource configuration for action find. Expected response to contain an array but got an object
Thanks again.
👍

[hans118]

And I can now this error in chrome dev console.

GET http://api.mydomain.com/v1/plans 404 (Not Found)
How to fix it?

[ataft]

thank you @sjmcdowall !

Once the size limit is dealt with, some apps may need to also set the parameterLimit to a larger size to avoid a "too many parameters" error. For example,

"body-parser#json": { "params" : { "limit" : "50mb", "parameterLimit": 1000000} },
"body-parser#urlencoded": {"params": { "limit" : "50mb", "extended": true, "parameterLimit": 1000000 }}

Or like this in server.js...

[youngtk96]

If you are hosting the server on NGINX try the following

Modify NGINX Configuration File

sudo nano /etc/nginx/nginx.conf
Search for this variable: client_max_body_size. If you find it, just increase its size to 100M, for example. If it doesn’t exist, then you can add it inside and at the end of http

client_max_body_size 100M;
Restart nginx to apply the changes.

sudo service nginx restart