Aws-sdk-js: DynamoDB.DocumentClient should support empty string properties
When the object contains a property with an empty string, it results in an exception instead of for example silently removing this property or replacing with a user-provided constant.
To be specific, the following example request:
var params = {
TableName: 'my-test-table',
Item: { Key:"foo", MyData: ""}
};
Results in the following error on DynamoDB:
[AWS dynamodb 400 0.588s 0 retries] putItem({ TableName: 'my-test-table',
Item: { Key: { S: 'foo' }, MyData: { S: '' } } })
Which is returned from the DocumentClient as:
Unable to store request. Error was {
"message": "One or more parameter values were invalid: An AttributeValue may not contain an empty string",
"code": "ValidationException",
"time": "2015-12-10T09:19:17.204Z",
"statusCode": 400,
"retryable": false,
"retryDelay": 0
}
thoean
All 174 comments
@thoean
What you're seeing is currently documented behavior:
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#putItem-property
String and Binary type attributes must have lengths greater than zero. Set type attributes cannot be empty. Requests with empty values will be rejected with a ValidationException exception.
This could be something we support in the future by having something like an ignoreEmptyStrings configuration option for the DocumentClient that was opt in.
chrisradek
on 17 Dec 2015
Your proposal sounds great and would solve my use case.
Whether opt-in or standard behavior depends how much DocumentClient is a simplification over DynamoDB, not only in simplification of providing the necessary parameters, but also trying to do the right thing for the client. I think this case has arguments in both directions, so opt-in to maintain compatibility to its current version makes sense.
thoean
on 17 Dec 2015
+1
RedaBenh
on 20 Mar 2016
+1
brcline
on 21 Apr 2016
+1
clicktravel-jaroslav
on 22 Apr 2016
+1
LakerBaker
on 9 Jun 2016
+1
hammadmlk
on 12 Jun 2016
+1
jesucarr
on 12 Jun 2016
+1
chandan449
on 23 Jun 2016
+1
simoami
on 23 Jun 2016
+1
itaykahana
on 26 Jun 2016
I worked around this by just wiping out the empty keys from the object:
var Item = { Key:"foo", MyData: "", Item: { Key:"foo", MyData: "", Item: { Key:"foo", MyData: ""}}};
function removeEmptyStringElements(obj) {
for (var prop in obj) {
if (typeof obj[prop] === 'object') {// dive deeper in
removeEmptyStringElements(obj[prop]);
} else if(obj[prop] === '') {// delete elements that are empty strings
delete obj[prop];
}
}
return obj;
}
removeEmptyStringElements(Item);
/*
{
"Key": "foo",
"Item": {
"Key": "foo",
"Item": {
"Key": "foo"
}
}
}
*/
bo01ean
on 29 Jun 2016
+1
pzimny
on 21 Jul 2016
+1
joshwils82
on 22 Aug 2016
Do we have support for empty strings in the latest version? If not, any timeline for this feature to be available? I have a specific case where I am persisting the payment response with hash. If I remove the keys with empty strings, the hash generation would fail later.
manikandants
on 28 Aug 2016
Hi @manikandants
Unfortunately, we can't share a timeline for this, but thanks for everyone who has given a +1 or voiced a concern about this feature request, as it helps us prioritize. I will look into this as soon as I can.
LiuJoyceC
on 29 Aug 2016
+1
John2496
on 1 Sep 2016
+1
ossie1971
on 1 Sep 2016
+1
lindell
on 2 Sep 2016
+1
joegesualdo
on 2 Sep 2016
+1
RhysC
on 12 Sep 2016
+1
bradledford
on 13 Sep 2016
+1
pavei
on 14 Sep 2016
+1
dustinbolton
on 18 Sep 2016
+1
amitatgit
on 19 Sep 2016
+1
jppellerin
on 22 Sep 2016
+1
cstavro
on 25 Sep 2016
+1
kpx-dev
on 25 Sep 2016
+1
iSkore
on 28 Sep 2016
+1
CJNE
on 28 Sep 2016
+1
tejasmanohar
on 6 Oct 2016
+1
Rajkumar55
on 7 Oct 2016
+1
danieldelouya
on 12 Oct 2016
+1
guillaumefont
on 13 Oct 2016
+1
seanchann
on 26 Oct 2016
+1
superandrew
on 27 Oct 2016
+1
rtrompier
on 28 Oct 2016
+1
On Fri, Oct 28, 2016 at 2:29 PM, rtrompier [email protected] wrote:
+1
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-js/issues/833#issuecomment-256908063, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACTfBJDHiiE36MoAsD9aENTITv7zoTfaks5q4eqMgaJpZM4GykZr
.
guillaumefont
on 28 Oct 2016
+1
whiteswift
on 2 Nov 2016
+1
csabakollar
on 2 Nov 2016
function cleanObj( o ) {
if( o && o === o )
if( typeof o === 'boolean' ) return o;
else if( typeof o === 'number' ) return o;
else if( typeof o === 'string' ) return o;
else if( Array.isArray( o ) ) {
let x = [], i = -1, l = o.length, r = 0;
while( ++i < l ) if( o[ i ] ) x[ r++ ] = cleanObj( o[ i ] );
return x;
} else if( typeof o === 'object' ) {
for( const k in o ) o[ k ] ? o[ k ] = cleanObj( o[ k ] ) : delete o[ k ];
return o;
} else
return 'Argument Error - Unknown Item';
}
+1
DianaIonita
on 8 Nov 2016
+1
Imran99
on 8 Nov 2016
+1
jamie-reddell
on 8 Nov 2016
+1
khoi-nguyen-2359
on 13 Nov 2016
+1
sasanala
on 16 Nov 2016
+1
jbuddha
on 17 Nov 2016
+1
Jorenm
on 19 Nov 2016
+1
alistairg
on 23 Nov 2016
+1
kellenp
on 30 Nov 2016
+1
asantibanez
on 3 Dec 2016
+1
cbarcenas
on 6 Dec 2016
+1
MerryOscar
on 6 Dec 2016
+1
nfmelendez
on 9 Dec 2016
+1, and at least be more specific in error message.
chaliy
on 12 Dec 2016
+1
bwestergard
on 12 Dec 2016
@LiuJoyceC could you disclose whether this is something that is actually being worked on or even discussed internally? I only ask because this bug seems to be almost a half-decade old. So is this something that actually could be fixed but hasn't, or is it a fundamental limitation of the DynamoDB internal structure?
jcready
on 13 Dec 2016
Glad to see something is being done, but that's really just trading one problem for a new one.
pzimny
on 16 Dec 2016
+1
enlightendata
on 18 Jan 2017
+1
brandonmbanks
on 23 Jan 2017
@brandonmbanks Converting empty values to null is now an opt-in feature of the document client; you can pass a boolean convertEmptyValues option to the document client constructor to have it do so. I'll open a PR to improve the documentation of this feature.
jeskew
on 23 Jan 2017
+1
svoeller99
on 8 Feb 2017
+1
njgrisafi
on 17 Apr 2017
+1
clearly
on 28 Apr 2017
+1
pixelfuture
on 2 May 2017
+1
ntrivix
on 10 May 2017
+1
ruleechen
on 15 May 2017
+1
elioncho
on 19 May 2017
this is much needed, please...
yipcma
on 30 May 2017
@yipcma As noted above, this has been implemented as an opt-in feature. Set the convertEmptyValues constructor option to true when creating a document client.
jeskew
on 30 May 2017
@jeskew I think what people want is the ability to store and retrieve an empty string. All this provides is a way to convert empty strings to null which really isn't the same thing.
jcready
on 31 May 2017
People want to be able to store JSON documents in DynamoDB but right now we only have two options: 1) store the full JSON as a string in a single column, or 2) use convertEmptyValues and accept that you will not be able to use 100% of the JSON spec without data loss.
jcready
on 31 May 2017
Unfortunately, that isn't something that can be handled client-side without knowledge of the expected data schema.
If the SDK included a higher-level abstraction that knew which types certain fields were expected to be (like the AWS SDK for Java's DynamoDB Data Mapper or the Ruby Aws::Record), then the client could convert a null value to an empty string or empty buffer as appropriate. That's a reasonable feature request, but it's not something that could be built into the document client without introducing breaking changes.
jeskew
on 31 May 2017
+1 from me
+1 from Leanne
Centurion
on 23 Jun 2017
+1
lamon
on 27 Jun 2017
+1
jeremysuriel
on 15 Jul 2017
+1
jstradli
on 18 Jul 2017
+1
ashikkalavadiya
on 19 Jul 2017
+1000
rafiek
on 2 Aug 2017
ive never seen a issue desired like this one. Any roadmap to implement the possibility to store an empty string in dynamodb? should everyone move to another database? 🙏🏼
superandrew
on 2 Aug 2017
@superandrew We've implemented opt-in, client-side support for this feature via the convertEmptyValues flag as discussed above.
jeskew
on 2 Aug 2017
in my opinion, this is more of a client side patch for something that needs to be done on the dynamodb side. And, more importantly, and as someone else said before me, null is no replacement for an empty string. So if more and more people keeps asking for a more complete fix, I wouldn't call this issue "closed". But, of course, still my opinion.
superandrew
on 2 Aug 2017
Just wanted to remind everyone to please keep the discussion constructive.
@superandrew I understand what you mean, but the SDK is a client-side library, and the DynamoDB team does not monitor the issue queue of any AWS SDK. For a server-side fix, I recommend that you reach out directly to the DynamoDB team via their AWS Developer Forum or via a support ticket.
jeskew
on 2 Aug 2017
@jeskew you can accelerate this type of request to AWS Team where plenty of people is facing issue with kind of problems! Though its almost 2 years, Developer finds a way on their(client) side as well!! LET IT GO!
ashikkalavadiya
on 2 Aug 2017
@jeskew this bug is now more than half a decade old. No one seems to be getting satisfaction over there so they are voicing their concerns here. Perhaps AWS could open-source DynamoDB and put the code up on GitHub so we could create an issue there and/or make a PR which allows empty strings to be stored.
jcready
on 3 Aug 2017
+1
tsmiser
on 3 Aug 2017
+1
worsnupd
on 6 Aug 2017
+1
amistein
on 16 Aug 2017
Slight change to the code suggested by @bo01ean so that it tokenizes and detokenizes the empty string. It works for me but it's annoying that I need to do this to preserve the original JSON when using dynamodb.
const token = '~&*^AWSSTILLHASNOTFIXEDTHIS!!!~&*^';
const tokenizeEmptyStringElements = function (obj) {
for (var prop in obj) {
if (typeof obj[prop] === 'object') {// dive deeper in
tokenizeEmptyStringElements(obj[prop]);
} else if(obj[prop] === '') {
// tokenize elements that are empty strings
obj[prop] = token
}
}
return obj;
}
const detokenizeEmptyStringElements = function (obj) {
for (var prop in obj) {
if (typeof obj[prop] === 'object') {// dive deeper in
detokenizeEmptyStringElements(obj[prop]);
} else if(obj[prop] === token) {
// tokenize elements that are empty strings
obj[prop] = ''
}
}
return obj;
}
emaildanwilson
on 17 Aug 2017
+1
ngocketit
on 24 Aug 2017
+1
arvinkx
on 1 Sep 2017
AWS Error Type: client, AWS Error Message: One or more parameter values were invalid: An AttributeValue may not contain an empty string
Validate empty string with " " SPACE
svnlabs
on 20 Sep 2017
+1
MattTunny
on 2 Oct 2017
+1
arctair
on 10 Oct 2017
+1
eddyfabery
on 11 Oct 2017
+1
terge
on 17 Oct 2017
+1
vvemulak
on 24 Oct 2017
+1
bo-ora
on 2 Nov 2017
+8
bluepeter
on 4 Nov 2017
+1
franciscotfmc
on 8 Nov 2017
+1
JamesDorrian
on 8 Nov 2017
+1
rocedavor
on 13 Nov 2017
+1
kunalpatilabg
on 14 Nov 2017
+1
atulrawat
on 17 Nov 2017
+1
josepapaianni
on 17 Nov 2017
+1
esteban-uo
on 22 Nov 2017
this is the single most requested thing I saw in my life.
superandrew
on 22 Nov 2017
Hi @superandrew,
You might want to check out the DynamoDB Data Mapper for JavaScript, which is now in developer preview. That project allows you to define your table schema up front so that empty strings can be saved as a sigil value compatible with DynamoDB's data model (null) and then deserialized back to an empty string when the item is loaded.
If you don't want to define a schema and don't mind transforming empty strings and buffers into null, you can also use the convertEmptyValues option on the Document client as discussed above.
jeskew
on 22 Nov 2017
thanks @jeskew as others have pointed out, this is not a problem of the js sdk but of dynamodb itself. I was just noticing how many people continue to request a dynamodb feature here because it's not possible to do it nowhere else.
superandrew
on 23 Nov 2017
+1
adimoraret
on 25 Nov 2017
+1
M1chaelTran
on 29 Nov 2017
+1
jhoel
on 29 Nov 2017
+1
sowmitranalla
on 11 Dec 2017
lol
invisibleroads
on 11 Dec 2017
Trying to insert the data directly from a third party service which has almost 234 fields, some of them are empty. +1
mymail4blogs
on 12 Dec 2017
+1
a1exwang
on 12 Dec 2017
+1
Mnilko
on 19 Dec 2017
+1
stephenfowler
on 20 Dec 2017
+1
chrisburkejr
on 24 Dec 2017
+1
bvanloocke
on 2 Jan 2018
+1
yzhong52
on 7 Jan 2018
+1
rtatiparthy
on 8 Jan 2018
+1
XBeg9
on 10 Jan 2018
+1
frankjl
on 11 Jan 2018
+100000
nfons
on 16 Jan 2018
+1
mlensment
on 18 Jan 2018
+1
Kivylius
on 24 Jan 2018
+1
devalpcyb
on 29 Jan 2018
+9001
alopes
on 30 Jan 2018
+1
synergiclabs
on 31 Jan 2018
const dynamodb = new AWS.DynamoDB.DocumentClient({
convertEmptyValues: true,
});
gesposito
on 31 Jan 2018
convertEmptyValues changes them into null values not empty strings https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html
chrisburkejr
on 31 Jan 2018
@chrisburkejr Empty string values will be rejected by DynamoDB; null is the only emptiness sigil that can be persisted in a DynamoDB table. If a particular field must always be a string, you could try using a schema-driven abstraction like the DynamoDB DataMapper, which can transparently handle empty strings.
jeskew
on 31 Jan 2018
This has been an issue for 5 years and has virtually no response on the DynamoDb forum, which is why you're getting hassled about it here. Maybe it's possible for you to raise this internally with the DynamoDb team so they can at the very least jump into the forum thread and address it with the users?
nickcoad
on 5 Feb 2018
+1
Aristocles
on 19 Feb 2018
+1
raaone7
on 22 Feb 2018
+1
menkveldj
on 25 Feb 2018
convertEmptyValues: true
was enough to fix it for me.
fptavares
on 28 Feb 2018
+1
BetaMee
on 3 Mar 2018
+1
tobiaszheller
on 5 Mar 2018
+1
hcho112
on 8 Mar 2018
+1
chauhanprateek89
on 14 Mar 2018
@fptavares solution works just fine. It's documented here. Here is a quick example on usage:
module.exports.postback = (event, context, callback) => {
const body = qs.parse(event['body']) || {}
const params = {
TableName: "conversions",
Item: {
"gclid": body.gclid,
"leadid": body.leadid,
"affid": body.affid,
"campid": body.campid,
"cid": body.cid,
"tid": body.tid,
"s1": body.s1,
"s2": body.s2,
"s3": body.s3,
"s4": body.s4,
"s5": body.s5,
"price": body.price,
"udid": body.udid,
}
}
const dynamo = new AWS.DynamoDB.DocumentClient({
convertEmptyValues: true
})
dynamo.put(params, (err, data) => {
if (err) {
Raven.captureException(err)
return callback(err)
}
callback(null, {
statusCode: 200,
body: JSON.stringify({ "message": "success" })
})
})
}
As noted by @synergiclabs this solution will not store empty strings. Instead it will convert an empty string to null. If you require empty strings this solution is not for you.
JamesTheHacker
on 17 Mar 2018
@JamesTheHacker That is not a viable solution. The option convertEmptyValues converts a string into a NULL value. There are instances you really need EMPTY STRING and not a null data type.
In JavaScript, "" !== null;
synergiclabs
on 17 Mar 2018
@synergiclabs Of course, but in my instance and maybe for others, the solution is fine as I do not require empty strings to be stored. This was likely the decision amazon took too which is why they've provided no support.
JamesTheHacker
on 17 Mar 2018
+1
Hanslen
on 16 Apr 2018
+1
seckcoder
on 24 Apr 2018
+1
Antommyj
on 24 Apr 2018
+1
spolyak
on 25 Apr 2018
+1
jianingYu
on 26 Apr 2018
+1, we really need to persist empty string into database.
wowmama
on 28 Apr 2018
+1 ... read in some other issue that empty strings will be supported in DDB in 2018 .. any update when?
kushbhandari
on 30 Apr 2018
+1
edenriquez
on 3 May 2018
+1
whitechdevs
on 15 May 2018
+1
richard-gebbia
on 23 May 2018
+1
nejcr
on 5 Jun 2018
+1
kimthangatm
on 11 Jun 2018
+1
cahenyossi
on 11 Jun 2018
+1
keesvanlierop
on 13 Jun 2018
can someone explain why setting an empty string would ever be considered invalid? this behavior makes no sense, the issue is closed, and it seems clearly incorrect despite being documented.
smtheard
on 22 Jun 2018
+1
andreafalzetti
on 23 Jun 2018
+1
AidanConnelly
on 24 Jun 2018
+1
jchirschy
on 28 Jun 2018
+1
PavelSizov
on 29 Jun 2018
+1
dmhalejr
on 5 Jul 2018
+1
cmavromichalis
on 5 Jul 2018
+1
jeremyputeaux
on 9 Jul 2018
+1
bogdan-maksak
on 15 Jul 2018
+1
onrul
on 16 Jul 2018
@chrisradek , may I ask you to use the lock conversation feature to avoid having another million of +1s and notifying so many users? The DocumentClient solves this already with the convertEmptyValues: true option. And if the request is to store empty strings in dynamoDB, then the client library is the wrong repository to report this.
Thanks.
thoean
on 16 Jul 2018
Just to be clear, convertEmptyValues does NOT solve the problem of dynamodb's inability to store empty strings. @thoean where is the correct place to report this issue if this isn't it?
richard-gebbia
on 16 Jul 2018
@richard-gebbia I don't know the correct place. But it's not the client library as stated many times above. Best done via your account manager at AWS, or if your at a smaller contract probably just by filing a support ticket from your AWS account.
thoean
on 17 Jul 2018
+1
This issue has been created on AWS forum for more than 6 years : https://forums.aws.amazon.com/thread.jspa?threadID=90137
rqton
on 26 Jul 2018
Locking this thread. This is a service behavior that can not be addressed by the SDK. Locking to prevent continued spam.
chrisradek
on 27 Jul 2018
Related issues
pihish
·
4Comments
kennu
·
3Comments
ataraxus
·
4Comments
ddo88
·
3Comments
lreher
·
3Comments
Most helpful comment
I worked around this by just wiping out the empty keys from the object: