Amplify-ios: Amplify Predictions identify not returning form(keyValue) and table data
Describe the bug
We are using amplify for our app development and one of the prediction API call is not working properly. It is working okay for Android but not for iOS.
https://docs.amplify.aws/lib/predictions/identify-text/q/platform/ios#identify-text-in-a-document
The API is not returning forms and table data in result?
We tried detectText(.form), detectText(.table), detectText(.all). Results are same for all of them.
To Reproduce
Steps to reproduce the behavior:
func detectText(_ image: URL, result:@escaping FlutterResult) {
_ = Amplify.Predictions.identify(type: .detectText(.all), image: image, options: PredictionsIdentifyRequest.Options(), listener: { (event) in
switch event {
case .success(let resultIdentity):
let data = resultIdentity as! IdentifyDocumentTextResult
print(data.fullText)
result(data.fullText)
case .failure(let error):
print(error)
result(error)
}
})
}
data variable contains
Expected behavior
tables and keyValues should contain data as it work in Android.
Environment(please complete the following information):
- Amplify Framework Version: 1.0.2
- Dependency Manager: Cocoapods
- Swift Version : 5.0
Device Information (please complete the following information):
- Device: iPhone8 plus, Simulator
- iOS Version: iOS 13
- Specific to simulators:
Additional context
No any other context about the problem.
vdvpie22
All 8 comments
Hi @vdvpie22,
Thanks for reaching out. It sounds like you are reporting a difference in the iOS version vs what Android offers. Can you please explain what you are seeing on Android, and how that is different on iOS? Specifically, how does the use case work for you on Android? How does it not work for you on iOS?
Thank you
wooj2
on 14 Jul 2020
Response with same document:
in Android (as expected):
{
"fullText": "INVOICE......",
"keyValues": [
{
"box": {
"bottom": 0.33146378,
"left": 0.7483453,
"right": 0.8464649,
"top": 0.32305497
},
"polygon": {
"points": [
{
"x": 0.7483453,
"y": 0.32305497
},
{
"x": 0.8464648,
"y": 0.32305497
},
{
"x": 0.8464648,
"y": 0.33146378
},
{
"x": 0.7483453,
"y": 0.33146378
}
]
},
"confidence": 85.76523,
"id": "64d01edb-0dce-4722-8d09-b2ed749dd0c1",
"value": {
"first": "DESPATCH NOTE",
"second": "S538796"
}
},
......
],
"lines": [
{
"page": 0,
"box": {
"bottom": 0.087794684,
"left": 0.86288506,
"right": 0.9639127,
"top": 0.072275944
},
"polygon": {
"points": [
{
"x": 0.86288506,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.087794684
},
{
"x": 0.86288506,
"y": 0.087794684
}
]
},
"confidence": 99.8893,
"id": "2a7531db-9649-408e-88cb-41b7f61d64eb",
"value": "INVOICE"
},
......
],
"rawLineText": [
"INVOICE",
.....
],
"selections": [],
"tables": [
{
"columnSize": 7,
"rowSize": 11,
"box": {
"bottom": 0.712523,
"left": 0.0673983,
"right": 0.9643701,
"top": 0.36833143
},
"polygon": {
"points": [
{
"x": 0.0673983,
"y": 0.36833143
},
{
"x": 0.9643701,
"y": 0.36833143
},
{
"x": 0.9643701,
"y": 0.7125229
},
{
"x": 0.0673983,
"y": 0.7125229
}
]
},
"confidence": 99.95759,
"id": "2f80525c-3328-48f5-892e-a9cc2c956dac",
"value": [
{
"column": 0,
"row": 0,
"selected": false,
"box": {
"bottom": 0.38292885,
"left": 0.0673983,
"right": 0.14748508,
"top": 0.36833143
},
"polygon": {
"points": [
{
"x": 0.0673983,
"y": 0.36833143
},
{
"x": 0.14748506,
"y": 0.36833143
},
{
"x": 0.14748506,
"y": 0.38292885
},
{
"x": 0.0673983,
"y": 0.38292885
}
]
},
"confidence": 99.95759,
"id": "4ff87dd2-0187-4232-a929-3fd86bc00969",
"value": "PRODUCT CODE"
},
....
]
},
{
"columnSize": 7,
"rowSize": 2,
"box": {
"bottom": 0.36267745,
"left": 0.045876656,
"right": 0.94214964,
"top": 0.31708455
},
"polygon": {
"points": [
{
"x": 0.045876656,
"y": 0.31708455
},
{
"x": 0.94214964,
"y": 0.31708455
},
{
"x": 0.94214964,
"y": 0.36267748
},
{
"x": 0.045876656,
"y": 0.36267748
}
]
},
"confidence": 99.65584,
"id": "3de1a322-46c5-4b88-9fdf-005b3d73e4eb",
"value": [
{
"column": 0,
"row": 0,
"selected": false,
"box": {
"bottom": 0.33703145,
"left": 0.045876656,
"right": 0.1599114,
"top": 0.31708455
},
"polygon": {
"points": [
{
"x": 0.045876656,
"y": 0.31708455
},
{
"x": 0.1599114,
"y": 0.31708455
},
{
"x": 0.1599114,
"y": 0.33703145
},
{
"x": 0.045876656,
"y": 0.33703145
}
]
},
"confidence": 99.65584,
"id": "7566fd5c-0d2c-4cb3-b82e-d18ae102082c",
"value": "INVOICE No"
},
....
]
}
],
"words": [
{
"page": 0,
"box": {
"bottom": 0.087794684,
"left": 0.86288506,
"right": 0.9639127,
"top": 0.072275944
},
"polygon": {
"points": [
{
"x": 0.86288506,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.087794684
},
{
"x": 0.86288506,
"y": 0.087794684
}
]
},
"confidence": 99.8893,
"id": "fcdda7c6-7f69-4d37-9cce-0ad348aa3075",
"value": "INVOICE"
},
.....
]
}
In IOS:
{
"fullText": "INVOICE......",
"keyValues": [],
"lines": [
{
"page": 0,
"box": {
"bottom": 0.087794684,
"left": 0.86288506,
"right": 0.9639127,
"top": 0.072275944
},
"polygon": {
"points": [
{
"x": 0.86288506,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.087794684
},
{
"x": 0.86288506,
"y": 0.087794684
}
]
},
"confidence": 99.8893,
"id": "2a7531db-9649-408e-88cb-41b7f61d64eb",
"value": "INVOICE"
},
......
],
"rawLineText": [
"INVOICE",
.....
],
"selections": [],
"tables": [],
"words": [
{
"page": 0,
"box": {
"bottom": 0.087794684,
"left": 0.86288506,
"right": 0.9639127,
"top": 0.072275944
},
"polygon": {
"points": [
{
"x": 0.86288506,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.072275944
},
{
"x": 0.9639127,
"y": 0.087794684
},
{
"x": 0.86288506,
"y": 0.087794684
}
]
},
"confidence": 99.8893,
"id": "fcdda7c6-7f69-4d37-9cce-0ad348aa3075",
"value": "INVOICE"
},
.....
]
}
That is, the IOS version does not return keyValue and tables as indicated in the first post
vdvpie22
on 15 Jul 2020
@vdvpie22
Thank you so much for the detailed report! Having real data made it really clear for me to dive into this.
This looks to be a bug in our code.
Specifically, our code which is parsing the response from Textract:
https://github.com/aws-amplify/amplify-ios/blob/eb5b7ff98ae9ac090dc8c71e374b8f3bcb0ee472/AmplifyPlugins/Predictions/AWSPredictionsPlugin/Support/Utils/IdentifyTextResultTransformers.swift#L69
If an element is of type .table or .keyValueSet, it may not have a value for block.text, and hence if we are parsing a .table or .keyValueSet, then we'll continue on line 70, thereby skipping that block! :(
Just to be clear about repro steps:
- Setup amplify predictions w/ identify enabled
- Use the following code ( or something similar)
override func viewDidLoad() {
super.viewDidLoad()
let url = URL(string: "https://raw.githubusercontent.com/aws-samples/amazon-textract-code-samples/master/src-csharp/test-files/employmentapp.png")
if let imageURL = url {
detectText(imageURL)
}
}
func detectText(_ image: URL) {
_ = Amplify.Predictions.identify(type: .detectText(.all), image: image) { event in
switch event {
case let .success(result):
let data = result as! IdentifyDocumentTextResult
print(data.keyValues) // Should have values
print(data.tables) //should also have values
case let .failure(error):
print(error)
}
}
}
Expected behavior
The above code should print out values for data.keyValues and data.tables
For reference, here is how the android code is processing the blocks:
https://github.com/aws-amplify/amplify-android/blob/71aa76d145812170d2f9d8e3110ef907427e08ef/aws-predictions/src/main/java/com/amplifyframework/predictions/aws/service/AWSTextractService.java#L132
wooj2
on 17 Jul 2020
Hello guys.
I think I am hitting the same issue. It this fix planned for next release? It still say 1.0.5 as a milestone but it is not there.
Thank you.
pavelrojkovtrepp
on 27 Jul 2020
Hi, @pavelrojkovtrepp. Sorry that this bug takes longer than I expected. I need more time solving this issue. And since I am already working on it. It shouldn't take long to release the fix.
ruiguoamz
on 27 Jul 2020
Hi, any updates?
vdvpie22
on 3 Aug 2020
Reference to the PR which is in progress. https://github.com/aws-amplify/amplify-ios/pull/660
ruiguoamz
on 3 Aug 2020
@vdvpie22 sorry for the delay, this fix has been released as part of Amplify 1.1.0
wooj2
on 13 Aug 2020
Related issues
Kilo-Loco
路
6Comments
Tom-RulesCube
路
7Comments
royjit
路
9Comments
YanRei
路
6Comments
Jun711
路
5Comments