Amphtml: Send amp-analytics data in post body

[emphaticsunshine]

This issue is about current implementation of amp-analytics. It is not any browser related. AMP provides 3 different ways to send data with amp-analytics:

• JS Beacon
• XHRPOST
• Image

It works fine with image as all the data gets appended as query param. But with xhrpost and beacon, there is no way to send data as post body. Whats the point of having xhrpost if data can't be sent as post body?

[aghassemi]

/to @rudygalfi @avimehta

[avimehta]

I think this is a valid FR. Let's look into this when we are overhauling requests.

[MicahMartin]

Any updates on this FR?

[nickveenhof]

I'd love to see this too. Our analytics system is expecting a POST payload instead of having all parameters in the url of the GET or POST request

[avimehta]

@MicahMartin and @nickveenhof Can you give examples on how you expect the data to be formatted in the post body? Is it JSON format or the normal query-param format or something else?

[nickveenhof]

Sure!

The API we have available is of the format as seen below. Obviously this format can change depending on the data that can be sent along. Our system allows multiple analytic "captures" to be sent in a single call, as this was invented before the webworkers that support beacon requests existed. So when a page was halted before it was sent, we keep the info in localstorage or cookie so it can be sent on the next pageview. This is irrelevant for Google AMP but since Google AMP does not allow custom POST bodies or any POST body at all we have to change our API to be completely GET compatible. As you can see, with the above use case, this is not so easy.

The request would be a complete custom POST body format in JSON format, similarly as to how complete custom GET request URLs can be built now.

{
    "touch_identity": "eGlhb2dlbGFpd2FuYQ==",
    "identity": "[email protected]",
    "identity_source": "email"
    "captures": [
        {
            "event_name": "Content View",
            "event_source": "web",
            "event_date": "2016-01-22 10:20:21:1",
            "content_title": "Page Title",
            "platform": "Windows",
            "person_udf50": "c3RvcHRyeWluZw==",
            ....
        }
    ]
}

I hope this clarifies it a bit?

[avimehta]

I think this use-case sounds fair. The problem I see is related to expansion of values in the object structure. Currently, all variables are either strings with variables in ${variableName} format. With this request, we'll have to traverse through the object and expand each of the values(/keys?). Actually making the data to be sent in post body is fairly simple. @nickveenhof @emphaticsunshine do you guys have bandwidth to implement the variable expansion of objects as per your needs? If not, someone from the AMP team will get to this but not sure exactly when.

@rudygalfi any ideas on priority/scheduling for this?

[rudygalfi]

This isn't high on our own stack ranking right now, but we'd very much welcome a PR to add this capability.

[nickveenhof]

We started working on making our basic analytics to work with GET requests, as we had no understanding of the work required to get this change in. Making the actual technical change is one thing, getting it in so that we can benefit from it in production is a whole other beast I assume :-)

[avimehta]

also, if you are willing to risk it, you can serialize the json object, use the variable replacement on that. The request will look something like this:

%7B%22touch_identity%22%3A%22${someVariable}%22%2C%22identity%22%3A%22${otherVariable}%22%2C%22identity_source%22%3A%22email%22%22captures%22%3A%5B%7B%22event_name%22%3A%22ContentView%22%2C%22event_source%22%3A%22web%22%2C%22event_date%22%3A%22${timeStamp}%22%2C%22content_title%22%3A%22${title}%22%2C%22platform%22%3A%22AMP%22%2C%22person_udf50%22%3A%22${clientId(foo)}%22%7D%5D%7D

the above corresponds to:
{"touch_identity":"${someVariable}","identity":"${otherVariable}","identity_source":"email""captures":[{"event_name":"ContentView","event_source":"web","event_date":"${timeStamp}","content_title":"${title}","platform":"AMP","person_udf50":"${clientId(foo)}"}]}

Again, there is no way to send the data in post body but this will be substatially less work for you to do on your systems. Once POST is fixed to handle this, you can switch to the POST version.

[oliverfernandez]

@rudygalfi @avimehta
I'm really interested in this, in order to send metrics to Elastic Search, that are later consumed using Kibana. This case also needs a POST request with body with JSON format.

I'd like to take this development. The approach I was thinking is to allow a json variable in AMP-ANALYTICS with the serialized JSON object we want to send. This approach is already used by the data-json attribute of AMP-AD

It would look like this:

json { "vars": { "json": "...", }, "transport": { "xhrpost": true } }

When this combination is present, then Transport. sendRequestUsingXhr would send the value of the json variable.

Does this make sense?

[lannka]

After discussions in #15729, we decided

• Instead of introducing new body payload object, we will reuse extraUrlParams.
• We will introduce a new boolean flag (useBody) in transport config to enable this feature.

Once useBody=true, given the available transport option on the client, runtime will decide whether to send extraUrlParams in the POST body, or as URL params. Despite the inaccurate naming, the advantage is to not miss any data in any transport options.

This work requires some refactoring of existing code structure. Refactoring WIP:

17885 #17892 #17934 #17933

[lannka]

Hi folks,

This feature is out in canary right now. Please play with it, and let me know how it fits your need.
To use it, simply put useBody in the transport config, and data in extraUrlParams:

"transport": {
    "beacon": true,
    "xhrpost": true,
    "useBody": true
},
"extraUrlParams": {
    "r": ${sourceUrl},
    "t": "${title}"
}

One caveat to keep in mind:
The value in the body payload are URL encoded (which should not). #19126
It's already fixed. We plan to patch it to canary soon.