Prebid.js: Drop of impressions on SSPs & adserver requests migration 1.X

[Deimos01]

Type of issue

Question to people who did the migration from Prebid 0.X to 1.X.
Are you facing a drop of impression requests with some SSPs (for the same inventory size) ?
Are you facing a drop of adcalls to your adserver with the new Prebid (for the same inventory size) ?

Description

On some SSPs we are seeing a significatif drop of impressions requested for the same inventory size:
-50% on Sovrn, -40% on Indexexchange,-33% on One by Aol, -20% on Smartadserver, -10% on Appnexus ...
And what is really weird, the adserver calls are dropping too for our network (-25% to -50%). We have a bidderTimeout at 2000ms and a "Master timeout" for the adserver calls at 2300 ms which means that all the calls to our adserver are automatically triggered after the Master timeout, with or without a Prebid winner.

We are using Prebid.js since the version 0.4.0 and it's the first time we are experiencing a large drop of adserver requests because of a new Prebid version. We did hundreds of different tests during the last 10 days, and it's not coming from a specific device, browser, geo or internet connection speed ... It's not a problem of adapter codes because some adapters are compatible with 1.X version since a while. But we are not able to catch a bug with the new version, everything seems to be working well during our tests.
It seems to be a global issue related to new Prebid core because the only thing which is working is to rollback on our 0.34 version. It's not related to GPDR or the ConsentModule also.
So I'm wondering if there is a deep issue with the new version of Prebid during the execution time ? or a really vicious bug hard to catch ?

Steps to reproduce

Test page is useless, the code execution is working well. It's just a open question to all the publishers.

Expected results

No impressions or adserver calls drop

Actual results

Impressions & adserver calls drop

Platform details

All browsers, all geos, all devices, 1.11 & 1.12 Prebid version. SmartAdserver as adserver.

Thank you,

[GosuDari]

Hey we switched over to 1.11.0/1.12.0 for gpdr suport as well and tested a lot and didn't see any issues until we rolled the changes out. We see like Deimos01 wrote the same issues with drop of ssps and even ad server calls. We have for now reverted to the 0.34.x version, hope the issue will be found and resolved soon, but its hard to find out what causes the issue

[YOzaz]

Happening here as well... still investigating.

[Saigredan]

We are facing similar issues. Make sure your bidder configurations are valid for current Prebid release. It doesn't solve the problem entirely though.

[jaiminpanchal27]

@Deimos01 Can you share some url's where we can debug ?

[Deimos01]

@jaiminpanchal27 Sure : Demo Page 1.12. Do not hesitate if you need more ressources (adserver reports, SSPs reports or other testpages).
@Saigredan our bidder configurations are valid. It's a global issue.
In the meantime, we decided to rollback the prebid version to 0.34 on our network.

Thank you,

[sultanboss]

Having the same issue. It's only filling a small percentage of placements and has a huge revenue impact.

Yes, this is a global issue.

[jaiminpanchal27]

@Deimos01 @YOzaz @sultanboss
I have pushed a fix for targeting which might have caused this issue.
Can you guys create a prebid build using branch https://github.com/prebid/Prebid.js/tree/getAdservertargeting-bugfix and test it if it solves the issue.

Let me know if you need any help with building prebid.js with this branch.

[Deimos01]

Thanks @jaiminpanchal27
That new Prebid version is now live on one of our websites.
I will keep you informed tomorrow.

[dcaminer]

@jaiminpanchal27 am I getting it right that the problem you fixed occurred only on the subsequent auction on the same unit? I.e. for refresh impressions or on websites using single page applications?

We've seen a major drop of impressions but we're pretty sure that the problem affects only non-refresh auctions. Moreover, it doesn't affect only prebid. DFP's reporting showed a significant decrease in the number of impressions but also in "total code served count". We also ran some a/b tests and apparently, the problem manifests itself only when there are a significant number of units and bidders on the page. We've tried limiting the number of units or the number of bidders, and both approaches worked. The "total code served count" and monetized impression as reported by DFP were the same in both variations.

It would seem to me that there is some problem with performance in the new version of prebid, that manifests itself only when there are many banners & bidders on the page.

@Deimos01 @YOzaz @sultanboss can you perhaps share some more details on home banners you have on the page and how many bidders you have in your setup? For us, it's 7 units and 6 bidders.

[jaiminpanchal27]

@dcaminer Yes you are right.
To add more here, some publishers do not use pbjs.setTargetingForGPTAsync function but set targeting with custom code and use pbjs.getAdserverTargeting function. I fixed the bug with pbjs.getAdserverTargeting in that branch. If results are good than will create PR and merge.

Thanks for sharing the results of you a/b tests. I haven't looked at the scenario you explained above.
Can you share your url where you have significant number of units and bidders ?

[YOzaz]

Hi guys. In our case, amount of requests is pretty much the same - maybe because we're using 100% of lazy loaded ads, and GPT async mode.
However, bid level is catastrophic - e.g. Sharethrough went down from ~50% to ~2%. We suspect that's something to do with generated ConsentString (GDPR), but nothing sure yet.

[Deimos01]

@jaiminpanchal27 Sorry but the new Prebid version with the fix on pbjs.getAdserverTargeting function is not the origin of our problem. The bug is still here.

To do a quick recap of our investigation :

• The problem has been noted on the following versions of Prebid : 1.3, 1.7, 1.11 , 1.12, 1.13 and 1.14-pre from @jaiminpanchal27. We didn't try all the other 1.X versions.
• It affects only non-refresh auctions (@dcaminer) but it seems not to be linked with the numbers of Adunits or bidders on the page. We are facing the problem with websites with only 2 Adunits et 3 bidders and also for websites with 6 adunits and 7 bidders.
• It's not linked to the ConsentModule or GDPR because we are testing Prebid versions without the ConsentModule and since 05/19/2018
• We are not able to find any errors in the console from Prebid when we are debbuging the websites
• And what is really interesting, it affects websites more than others. Websites with top geo-audiences from South America, Africa or Asia are really more impacted than websites with North America or Europe top geo-audiences.

So we are thinking that issue is coming from the new way to call the bidders in the 1.X Prebid versions, it seems to be slower for some geos. So maybe we are reaching the timeout more often than with the legacy version ?
That would explain the drop of impressions for some SSP and also the drop of adserver requests ( the visitor leave the page before the adserver call because SSP are too long to answer). It's just an idea, what do you think guys ?

[jaiminpanchal27]

Have been testing with @Deimos01 page with both legacy and 1.x by throttling network.
On Slow 3G/Fast 3g
In 1.x: See lot of xhr requests getting cancelled after 1st auction is completed.
In legacy: Its working fine.

The reason for this can be :
In 1.x we added a timeout on xhr. timeout value will be the value passed in requestObj http://prebid.org/dev-docs/publisher-api-reference.html#module_pbjs.requestBids or if that is not passed we use bidderTimeout. We did this because in 1.x we are not adding a no bid response to the auction and hence no reason to continue the xhr request if it responds after timeout.

1.x has a concept of concurrent auctions, you may have too many xhr requests triggered at the same time if you have significant number of auctions running at the same time. It also depends on number of adunits/bidders configured. Browsers can at max only handle 6 requests at at time. See Here

@dcaminer did some a/b tests and observed same. https://github.com/prebid/Prebid.js/issues/2648#issuecomment-394835164

Also based on last comment by @Deimos01 he is facing drop in geos with slowers internet connections. This will make xhr request fail.

Increasing bidderTimeout for now may solve this problem.

[jaiminpanchal27]

@Deimos01 What timeout are you using ? Can you increase and test ?

[dcaminer]

@jaiminpanchal27

1. On the initial load, we run a single auction. I.e. we do single requestBids call and pass all the units as the parameter. I don't think we're running concurrent auctions
2. I'm pretty sure prebid was dropping slow requests prior 1.x https://github.com/prebid/Prebid.js/blob/legacy/src/ajax.js#L86 But prior 1.x it did this silently, without any notifications. Hence the difference you're seeing. Ofc in 0.34 some bidders could send requests w/o using ajax module, but still, only those responses that were received prior t/o were sent to the ad server. AFAIK Prebid 1.x introduced no changes in this regard, the only exception is that now all requests are sent by Prebid and not by adapters.
3. There are certain limitations that slow down bid responses as more units/bidders are added. But the same limitations apply to 0.34 as well. If in Prebid 1.x it takes more time for bidders to respond that should be considered as a bug. Raising timeout is not the solution, as it has obvious downsides.

[Deimos01]

@jaiminpanchal27 as I said in my initial post, we have a 2000ms bidderTimeout (which is very long). We tried to push the bidderTimeout to 3000ms (which is insane) without seeing significant improvements. And it's not the solution. How the same endpoint URL of a SSP could answer slower depending of the Prebid version ?

The first reason (timeout on xhr) seems legit but it shouldn't impact the adserver requests.

But what you are saying about "concept of concurrent auctions" is really interesting.
It means that if there are more than 2 adUnits with 3 bidders configured on the page (+6 xhr requests), we will face latency because of that "new concept of concurrent auctions" ?
What is the purpose of that new improvement ?

Thanks,

[jaiminpanchal27]

@dcaminer I did not mean raising timeout is the permanent solution. I agree its a bug and we are debugging in depth as we speak.

@Deimos01 You won't face latency with just 2 adUnits and 3 bidders. I don't have a test page to show stalling of requests.

I agree 3000ms is very very high. Are you also testing on this https://www.themoneytizer.com/demo/prebid_timeout?pbjs_debug=true page

Thanks for helping out here. One last thing i have for now is building prebid with https://github.com/prebid/Prebid.js/blob/master/modules/pre1api.js

[dbemiller]

How the same endpoint URL of a SSP could answer slower depending of the Prebid version ?

There's a subtlety here which I don't think is coming across very well in the comments...

Consider the following scenario:

• The browser/device allows at most 6 concurrent XHR connections
• You want to run two auctions, both with 6 bidders and a timeout of 2000ms
• In the first auction, 1 bidder returns in 100ms, and the rest return in 1900ms.

In both versions of Prebid, the auction timer starts when the first XHR call executes.

In Prebid 0.x, the first auction must complete before the second one starts. The second auction starts at 1900ms, and every bidder in both auctions returns within the timeout.

In Prebid 1.x, Prebid _tries_ to start all the adapters from all the auctions at once. However, the browser queues XHR calls because it only allows 6 concurrent connections. When the first adapter from the first auction returns (100ms), _an_ adapter from the _second_ auction is allowed to make a call.

In this case, the second auction's timer starts at 100ms, when the first XHR call in that auction executes. Since the first auction continues to use 5 connections for 1800ms more, the other adapters in the second auction only get 100ms to complete before the auction ends.

This wouldn't affect @dcaminer, since they're only running a single auction... but it could definitely increase the timeout rate for anyone using concurrent auctions.

[YOzaz]

We've just pushed a manual queuing system at our end to avoid concurrent Prebid.js auctions. This should basically simulate pre-1.0 auction system.
Will leave it running for a few days and will come back with results.
I was seeing "xhr cancelled" messages in console before, not anymore - but still seeing some "xhr timeout after 500ms", while prebid config is set to have a timeout of 2000ms. Weird.

[jaiminpanchal27]

@YOzaz You can use pre1api module to enable legacy features with 1.x

[Deimos01]

@jaiminpanchal27 the 1.12.0 Prebid version with the pre1api module activated is now live on one of our websites.
The test page is updated too.
I will keep you informed tomorrow.

[Deimos01]

@jaiminpanchal27 I'm really sorry (again) but the Prebid version with the pre1api module activated is not fixing our problem. We have exactly the same problem & metrics as we had with the 1.11 - 1.12 - 1.13 - 1.14 Prebid version ...

[gramorris]

No problems across 10 sites here. We do have an auction queue in place that we've retained from pre-1.0 so if it is a concurrent auction issue we wouldn't experience it.

[pdramos1]

@gramorris when you say an auction queue what do you mean? Are you saying you are using the pre1api module? Or, you have something specific or some other method of queueing requests?

[gramorris]

@pdramos1 we have our own auction queue implemented outside of PreBid

[lebrunm]

Hello, I identified something similar last week because of the GDPR module: it appears the timeout of the module do not behave as usual timeouts but blocks everything (no Prebid request sent to SSPs etc). In other term, if your Prebid setup has a timeout of 1000ms while the GDPR timeout is set to 1000ms, you should see a massive drop of ad requests to SSPs (~90-100%).

I hope that helps ;)

[dmitriyshashkin]

@lebrunm perhaps you've set "allowAuctionWithoutConsent" to false. Then such behavior would be expected

[lebrunm]

@dmitriyshashkin No, "allowAuctionWithoutConsent" was set to true in my setup.

[dmitriyshashkin]

@lebrunm I'd recommend you to open a new issue. I'm pretty sure it is not supposed to work like this. I don't think it is related to the current issue, as Deimos01 mentioned that they tried to disable GDPR module altogether and saw no positive results.

[mercuryyy]

@jaiminpanchal27 Any Update on this?

[dbemiller]

I did stumble across some very... suspicious... code surrounding the end of auctions.

The bidderFactory generates an object with a callBids(..., done) function. However, it doesn't get called under certain circumstances.

In auction.js, the callback sent by the bidderFactory to the bidders increments a counter which is then used to figure out when the auction is over. It looks like auction.js tries to compensate for this... but that's a _really_ fragile design.

For example, the "fix" won't happen if bidResponse.videoCacheKey is true, which it will be if Rubicon bids are involved.

That's just one example that I see off the top of my head... but there may be many other logical discrepancies between the two too. It's simply not safe to write two logic branches in different parts of the code and require them to stay in sync.

My general suggestion would be to simplify the code which determines when auctions end, make sure it's written in a single file, write good tests for it, and see if the problem goes away.

[dbemiller]

One question for @Deimos01, @mercuryyy, @YOzaz, and anyone else interested in this.

Could you let us know:

1. Whether or not you're also bidding on Video.
2. Whether or not you're using Rubicon.
3. Whether or not your config defines a cache.url

That'll help us gauge how likely this is to be causing your problems.

[Deimos01]

@dbemiller we are also bidding on video but for a small amount of our websites. Same for Rubicon.
Anyway, all my previous tests were done on websites without video adunit & without Rubicon too

[pdramos1]

@dbemiller we are seeing it sans video over here at Sovrn as well.

[aprakash-sovrn]

@gramorris Can you possibly provide more details on your auction queue? Is that something that is shareable?

[GosuDari]

@dbemiller we see the drop as well and we we only deliver banner ads, in addition we refresh visible ads every 30 seconds

[mercuryyy]

@dbemiller we also only do banner ads and refresh every 60 sec

[snapwich]

@aprakash-sovrn the pre1api module includes auction queueing if you want to include that module to test and see if you have any improvements. My initial testing w/ @Deimos01's test page shows a sizable improvement but I'm still running tests.

[YOzaz]

We are calling Unruly instream video, but as a separate pbjs call.
Shifting to queued calls instead of letting them run async did improve things, but for about 10-20% only.

Yes, we're using Rubicon.
Our config does not define cache.url.

[aprakash-sovrn]

I thought @Deimos01 said the pre1api module didn't do the trick for them? We tried it as well and didn't see any improvement on our end. Other than including the module, was there anything else you had to changed in your setup?

[dbemiller]

We found at least one clear bug. This affects people who are using Prebid Server, and set their s2sconfig.timeout to be shorter than the arg.cbTimeout that you pass into newAuction. If your newAuction argument doesn't define a cbTimeout, it defaults to 3 seconds.

The adapterManager.makeBidRequests function defines bidRequest objects, and adds ones to Prebid Server to the array first. For PBS requests, bidRequest.timeout is defined by s2sconfig.timeout. For Client-side requests, bidRequest.timeout is defined by cbTimeout.

Later, the timeout on _the first bidRequest_ is used as the ajax timeout. This timeout is then used for _all_ Bidders, including the client-side ones.

---

tl;dr: @jaiminpanchal27 is fixing a bug which affects people using Prebid Server. In the meantime, PBS users should make sure that their s2sconfig.timeout is the same as the cbTimeout property in the object they pass into newAuction. If you don't define a cbTimeout, then set s2sconfig.timeout to 3000 (3 seconds).

[bretg]

Question for those affected by a 1.x performance issue.

The original post from @Deimos01 noted "On some SSPs we are seeing a significant drop of impressions -50% on Sovrn, -40% on Indexexchange,-33% on One by Aol, -20% on Smartadserver, -10% on Appnexus "

1) is this pattern consistent across other publishers?
2) are there bidders less affected or not affected at all?

Besides a platform-level analysis, we'd like to look into adapter changes like SRA.

[snapwich]

As @bretg just alluded to, one of the findings I found in my tests on an active page was that there was a significant drop in impressions for the appnexusBidAdapter when comparing 0.34 vs 1.x with a slow connection speed. However, that is a comparison between 0.34 appnexusBidAdapter vs 1.x appnexusBidAdapter and I was notified by appnexus that in the transition to 1.x the appnexusAstBidAdapter became what is now the appnexusBidAdapter, so it wasn't really a fair comparison (which is true). The AST adapter uses SRA while the old 0.34 appnexusBidAdapter does not.

In my tests, the SRA request in the appnexusAstAdapter performed much poorer on a slow internet connection as _all_ the bid requests had to come back in a single request before the timeout and it often didn't make it, whereas with single requests at least _some_ of the bids could make it before the timeout if not all of them could make it.

If you used appnexusAstBidAdapter in 0.34 before you upgraded to the appnexusBidAdapter in 1.x then this shouldn't cause a decrease in impressions for you. However, if you were using the appnexusBidAdapter in 0.34 before upgrading to 1.x and were subsequently upgraded from non-SRA to SRA, you may have seen a decrease in impressions.

[snapwich]

Also, a small difference I noticed in testing so far is the inclusion of a "last bid" in 0.34. In 0.34 the closure of the auction when timeouts were involved was a bit different and was refactored in 1.x with a 0.34 auction often allowing a last bid to come in after the timeout. This could _possibly_ result in an extra bid in in the case of the auction hitting timeout and targeting being sent _after_ the last bid is received. This can be seen in this performance timeline here:

I wouldn't consider this any smoking gun in impression drop though.

[YOzaz]

Additional messages we see when running with pbjs_debug=true. Messages below are not visible without debug mode enabled.

• SyntaxError: Unexpected end of JSON input at JSON.parse (<anonymous>) at Object.O.contentType [as success] (prebid-1.13.0-pre.js:11) at XMLHttpRequest.i.onreadystatechange (prebid-1.13.0-pre.js:3). This was happening on previous 1.x build as well. My guess is some of the adapters are not responding with valid JSON structure, but I'm not sure which one is.
• WARNING: module criteo is loading external JavaScript.

[YOzaz]

Update: it looks like a mixture of updates has fixed things at our side.
We are going to revert timeout settings, and update Prebid.js to 1.15.x, remove pre-api module and auction queuing and see if it'll stay as it is.

[dbemiller]

The new release fixes some significant bugs which were contributing to, if not totally causing, some of the problems here.

Would anyone mind testing it out and reporting back what you find?

[YOzaz]

@dbemiller it looks like it's working much better now, but we have a huge drop in Sharethrough requests now (others are fine). Trying to investigate what's so different about it.

[dcaminer]

We'd be glad to run a test, but IndexExchange still hasn't fixed their discrepancy issue. Until they do it any tests with 1.x branch would be too costly, unfortunately.

[Slind14]

Could always test it on a small portion of the traffic ;) (like one should always do with new versions)

[Deimos01]

Hello guys,

We are testing the 1.15 version for +24H now.
Unfortunately we are facing the same problem of massive drop of adserver requests.
Our testwebpage is updated if you want to check our setup : test page

[snapwich]

@Deimos01 I might be worthwhile turning off concurrency to see if your issues are concurrency related or if it's something else entirely.

http://prebid.org/dev-docs/publisher-api-reference.html#setConfig-Max-Requests-Per-Origin

You can turn off concurrency in 1.15 with pbjs.setConfig({ maxRequestsPerOrigin: 1 });

[snapwich]

@Deimos01 also I just noticed your test page includes indexExchange. Have you tried excluding them to see if it is IndexExchange specific and related to this issue? #2690

[YOzaz]

@snapwich we've tested with concurrency turned off an on, without any effect. Unfortunately.

[Deimos01]

@snapwich We have exactly the same results on our side, even with concurrency turned off.
We are aware of the IndexExchange issue so the SSP is excluded when we are testing on our network.

[YOzaz]

To everybody working with this - please inspect what warnings do you get with ?pbjs_debug=true in your console. We've noticed that user syncs were failing, increased syncsPerBidder to 50 and Sharethrough is back to normal.
Issue submitted here: https://github.com/prebid/Prebid.js/issues/2781.

[janvdc]

We're experiencing the same issue. We switched from 0.34 to 1.15 and took good note of our analytics before and after. Significant reduction in impressions with the same number of requests. Sample URL: https://www.ancient.eu/Black_Death/?pbjs_debug=true

[YOzaz]

@janvdc in your case, I see you're running auction before consent is gathered. Unfortunately, this won't work - you have to get that "consent" popup first, queue any pbjs.requestBids() calls, and process them only after popup is closed with whichever answer.
At the moment, you're running an auction without a consent given, so only returning visitors will get ads served.

[YOzaz]

Update: after rolling out current master as 1.16.0-pre, and logging GA events, we saw 20% timeout rate for all bidders.
Timeout setting was at 2s, increased to 3s. Also, now ensuring GDPR is fully loaded before pbjs.requestBids() calls are executed.
This increased bid rate a lot today. My guess would be something is wrong with all that __cmp() queued calling system, and timeout calculations - before, 2 seconds were more than enough for bidders to respond.

[janvdc]

@YOzaz thanks for pointing this out. I'll have a look into this!

[janvdc]

I had another look into this and I don't think the CMP is the reason. Impression rates are down across the board, not just in the EU:

US = 30% PB0.34, 26% PB1.15
UK = 19% PB0.34, 16% PB1.15
AU = 17% PB0.34, 15% PB1.15

[jaiminpanchal27]

@janvdc I see multiple versions of prebid loading on your page.

[YOzaz]

@jaiminpanchal27 I guess that's a prebid + postbid combination. As postbid usually is running in an iframe, this shouldn't be an issue.

[janvdc]

It's Kumma, one of our demand partners, who run Prebid inside their iFrame.

[Deimos01]

@janvdc @YOzaz Sorry but as I told before, this issue is not linked to any CMP system or the consentManagement module. We are facing the original problem (drop of adserver & SSP requests) for more than one month now, in all the 1.X Prebid versions (even before the release of the ConsentModule) and we are testing without any CMP & with the ConsentModule disabled.

[YOzaz]

@Deimos01 from which country IP we should test your link? Is it USA-based?

[janvdc]

Just wanting to report back on a few more tests. We saw a slight improvement with the following changes in Prebid 1.15:

• Increased timeout to 3 seconds (from 1 second)
• Set syncsPerBidder to 50 (from whatever the default is)

We've now made the following additional changes and will report back once we have enough data:

• set timeoutBuffer to 400ms (from 200ms)
• set maxRequestsPerOrigin to 1 (from 4) -- to emulate 0.34.x behaviour
• set enableSendAllBids to false (from true) -- to reduce the overall amount of data transferred

[janvdc]

Reporting back, we've managed to bring impressions and ad revenue back to original 0.34.x levels using 1.15. We've simply set the config as follows:

pbjs.setConfig({ userSync: { syncsPerBidder: 50 }, maxRequestsPerOrigin: 1, enableSendAllBids: false, timeoutBuffer: 400 });

I'm pretty sure that the maxRequestsPerOrigin: 1 setting is what stopped the impression loss, as @snapwich had suggested. I believe that the browser making too many requests to the same source causes timeouts or dropped requests.

We also have Sovrn CMP turned on (with non-cookied ads loading in the background) and the ConsentModule activated in Prebid, so I'm also pretty certain that the CMP has nothing to do with it.

[YOzaz]

@janvdc, as a test, would you mind setting enableSendAllBids to true? My additional guess would be that targeting is set incorrectly in concurrent auctions with this.

[janvdc]

@YOzaz I'll give that a try, yes.

I'm looking at this in more detail, and I now that we're just switching quarter it's hard to tell what's baseline and what is a quarterly drop of revenue, what is caused by our partners, and what's actually caused by Prebid 1.15. I'm not 100% sure the issue is fixed with the above. More testing needed....

[dmitriyshashkin]

@janvdc IMHO you should always use A/B tests if you can isolate the changes to one variation. Even mid-quarter you can see a sudden increase in revenue because some large buyer started to bid for your inventory across multiple SSPs. Or mb some changes in the website improved it's pages performance, or there is drop/increase of traffic from some particular location.

[janvdc]

@dmitriyshashkin Agreed! I'd love to do an A/B test to be sure. I'm not sure how I would be able to separate the different versions of Prebid in our analytics, though (Roxot Analytics). I'll think some more about that.

[Slind14]

@dmitriyshashkin Agreed! I'd love to do an A/B test to be sure. I'm not sure how I would be able to separate the different versions of Prebid in our analytics, though (Roxot Analytics). I'll think some more about that.

You could probably setup an additional website with a different id. If not I'm happy to hook you up with https://yield.assertcom.de/ where you can run multivariate tests. We are slightly affected by this as well with our O&O sites, so happy to help for a resolution.

[dmitriyshashkin]

@janvdc do you by chance use DFP? If so you can pass variation through googletag.setTargeting method and then break down your stats by "key-value" in DFP's reporting. I can provide you with more detailed instructions

[janvdc]

@Slind14 thanks for the tip! Assertcom looks like a great solution. If you could hook me up, that would be great: jan.vandercrabben [AT] ancient.eu

@dmitriyshashkin Yes we use DFP ad we could pass a KV in. Good idea. However, that would only give us information on bids won, not on all the requests made. The main problem is bids timing out (we seem to think), so the DFP report would tell us nothing about that, or would it? Let me know if I'm missing something here, I'm by far no DFP expert.

[Deimos01]

Hello guys,

We were running tests during the past week, with all the configurations mentioned in this issue and we are still facing a massive drop of adserver requests, even with the last configuration from @janvdc :
pbjs.setConfig({ userSync: { syncsPerBidder: 50 }, maxRequestsPerOrigin: 1, enableSendAllBids: false, timeoutBuffer: 400 }); and the 1.15 Prebid version.
Here some interesting metrics.
Visitors by country (1.15 compare to the 0.34):

• Brazil -65%
• Colombia -52%
• China - 75%
• Hong Kong -74%
• India -93%
• Indonesia -70%
• Japan - 76%
• South Korea -69%
• Pakistan : -54%
• Philippines -68%
• Serbia -64%
• Ukraine -90%
  No significant drop for countries like France, UK, USA, Germany, Spain and Italy.

So the real problem is definitely linked to the internet connection speed as @jaiminpanchal27 suggested one month ago but even with the concurrency system turned off and a 3 seconds timeout, we are still facing the original problem.

What is the solution ?
Because without an appropriate solution, a lot of international publishers and networks won't be able to do the 1.X migration and to use the ConsentModule.

[headerbidding]

Could it have something to do with the anomaly/bug observed (and not yet fixed) in: https://github.com/prebid/Prebid.js/issues/2677 ?

[Deimos01]

Thanks for the quick answer @headerbidding but I don't think the 2 problems are linked.
We are not using invalid bidder names during our tests and even if we did, I don't understand how your bug could affect some geos more than others ...

[headerbidding]

Just a gut feeling that in the code a timed out bidder could cause similar behavior to an invalid bidder...

[lebrunm]

We ran similar tests than @Deimos01 with "maxRequestsPerOrigin: 1, enableSendAllBids: false" (but with syncsPerBidder and timeoutBuffer at their default values), but we've not seen any improvements.

Since the issue might be due to the max. number of connections allowed by browsers, we though it would be interesting to get some metrics showing the evolution of Ad impressions over the Total Ad requests (Prebid v1.15 vs 0.34):

• Internet Explorer 11: -118% _(theoretical max conn. allowed: 13)_
• Firefox: -9% _(6)_
• Safari: -20% _(6)_
• Chrome: -30% _(6)_
• Microsoft Edge: -48% _(?)_

[jaiminpanchal27]

@lebrunm Can you share more details with us on [email protected] and [email protected]

[mkendall07]

hey all. Thanks for testing out the fixes we've put up so far! Would someone be comfortable sharing their test pages (both 1.x and 0.34.x versions) so we can take a look? Thanks!

[Deimos01]

Hello @mkendall07,
Here the test pages : 0.34.6 & 1.15.0
Thank you.

[janvdc]

Here you go:

https://www.ancient.eu/article/637/roman-daily-life/?pbjs_debug=true&country=us (0.34)
https://www.ancient.eu/article/637/roman-daily-life/?pbjs_debug=true&country=fr (1.15)

[headerbidding]

On the 1.15.0 test page, after the 3rd refresh I got an ad that was not rendering.
I also noticed that the "waiting for ..." is on a continuous loop:

Windows 10 - Chrome Browser - latest version - browsing from the US (Florida) - Frontier FIOS

[jaiminpanchal27]

Hi Everyone, Failed xhr requests is one of the problem so we have temporarily added config to disable ajax timeout in this branch https://github.com/prebid/Prebid.js/tree/disable-timeout

To use this,

pbjs.setConfig({
  disableAjaxTimeout: true // default value is false
})

Test using this new config and let us know if it improves anything.

Thanks to @lebrunm for helping out with more details on his setup.
We pulled up some reports for appnexus bidder and found that number of ad requests to appnexus decreased after switching to 1.x. I think that may be the case for other bidders as well. One of our other clients also reported around 20% drop in ad requests overall.

We are working on

• finding the reason for decrease in ad requests
• IE11 and edge have been showing more xhr request failures compared to other browsers so checking for that as well.

[Deimos01]

Hello @jaiminpanchal27 ,
We are testing your new version since 2 days and we are still facing the same issue of a massive adserver call drop.

[jaiminpanchal27]

Thanks @Deimos01 for trying this. We are in process of creating setup using appnexus platform. I will update you once we have the reports.

[mkendall07]

@Deimos01
Can you also email us the data you are seeing? https://github.com/prebid/Prebid.js/issues/2648#issuecomment-404191839

[mercuryyy]

@mkendall07 Looks like most of us having this issue are using s2s as well. It looks to me like when s2s is not returned on time even if client side bidders returned on time the action fails.
Unless you set bidderTimeout but then you lose more fill from s2s

[lebrunm]

On our side we've not yet tested "disableAjaxTimeout:true", but doing the following finally fixed the issue:

• (1) the use of "syncsPerBidder: 50" and "timeoutBuffer: 400", as suggested above
• (2) the use of the Prebid timeout instead of the usual JS function (setTimeout)

The (Prebid) revenue is now back to normal for 3 different sites (with international traffic). Thank you to everyone participating in this thread!

@mercuryyy : we're not using s2s, so you may experience a specific s2s issue?

[headerbidding]

@lebrunm: Can you explain for a less technical person "the use of the Prebid timeout instead of the usual JS function (setTimeout)"

[lebrunm]

@headerbidding There are now 2 ways to call your ad server when your Prebid timeout has been reached:

• (1) by using the standard JS function (setTimeout) as shown on this Getting started page
• (2) by using pbjs.requestBids({ timeout: YOUR_TIMEOUT }), which gives Prebid the control of when the ad server is called.

We've not tested an implementation with only "syncsPerBidder: 50" + "timeoutBuffer: 400" - it is likely the change of the Timeout logic had no incidence at all since @janvdc succeeded to fix the issue with the syncsPerBidder & timeoutBuffer params.

[YOzaz]

@lebrunm there is also a global config called bidderTimeout. Did you try playing around with it as well?

pbjs.setConfig({ bidderTimeout: 3000 });

[lebrunm]

@YOzaz No, we never tried the bidderTimeout param.

[mkendall07]

hey all.

If you still are having this issue can you please link your site, info about the setup and what you have tried to resolve it? It's odd that some folks are reporting success with various setting but others are not. So would like to look for some more patterns.

Feel free to email me directly as well.

[janvdc]

After all, I'm not 100% sure that we've managed to fix it on our side. It's a lot better, but I don't know if the revenue is back to 100% of 0.34. We're going to run a proper A/B test for a few days and I'll report back.

@lebrunm, could you please post a code example of how you're calling pbjs.requestBids()? I'm not clear on how it works in combination with pbjs.setTargetingForGPTAsync() and pbjs.initAdserverSet = true. Do we still need to call these separately, or is this handled by requestBids()? Thank you!

[lebrunm]

@janvdc The following piece of code from the Getting Started page is still required. Updating your implementation with the Prebid timeout can be done with "pbjs.requestBids({ timeout: YOUR_TIMEOUT })" as follows:

        pbjs.que.push(function() {
            pbjs.addAdUnits(adUnits);
            pbjs.requestBids({
                timeout: YOUR_TIMEOUT,
                bidsBackHandler: initAdserver
            });
        });

        function initAdserver() {
            if (pbjs.initAdserverSet) return;
            pbjs.initAdserverSet = true;
            googletag.cmd.push(function() {
                pbjs.que.push(function() {
                    pbjs.setTargetingForGPTAsync();
                    googletag.pubads().refresh();
                });
            });
        }

Again, I do know whether this helped to fix the current issue (drop of impressions) since - as said in my above comments - we released 2 major changes at the same time.

[janvdc]

@lebrunm Thanks for the more detailed explanation. We've just turned on an A/B test that also incorporates your fix. I'll report back when we got the results.

[janvdc]

Our A/B test is complete and I can say with confidence that all of the changes above applied together have fixed the issue. We are now earning more with 1.15 than with 0.34. The following are the results of our test, conducted on US users only:

pbjs_world.js => Prebid 0.34
pbjs_eu.js => Prebid 1.15

Many thanks to everyone in this thread who helped us figure it out. Special shoutout to @Slind14 for his awesome Assertive Yield analytics platform that helped us enormousely with the A/B test.

[mercuryyy]

@janvdc

Can you share the exact setup you used for pbjs_eu.js ?
Can you also run a test on pb 1.19.0 ?

[janvdc]

@mercuryyy

Sure, you can find our configs here:

1.15 => https://www.ancient.eu/js/pbjs_eu.js?v=1534296564
0.34 => https://www.ancient.eu/js/pbjs_world.js?v=1534296564

We're going to upgrade to 0.19 soon, and see if that makes any additional difference.

[Deimos01]

Hello guys,

I don't want to kill the mood but the bug isn't resolved.
As I said before, some geos are more affected than others.
We did a test with the setup of @lebrunm + 1.19 Prebid version and we are still facing a massive drop of impressions for :

• all south american countries (from -90% to -60%)
• India (-90%)
• Indonesia (-80%)
• Malaysia (-80%)
• Philippines (-90%)

And probably other exotic countries are impacted.
@janvdc & @lebrunm, could you please share some geo/internet speed reports ?

Thank you

[lebrunm]

@Deimos01 I could share some data with you on Monday. Can you please confirm you're still not using the Prebid GDPR module?

[Deimos01]

@lebrunm Many thanks for the prompt reply and yes the GDPR module is not activated and not linked to this issue.

[janvdc]

@Deimos01 We've only A/B tested in the US so far. We could now include other countries, too. I'll report back to you as soon as I know more.

[lebrunm]

@Deimos01 Below is the evolution of the the Prebid impression ratio per country (AI / AR):

_(with Prebid 0.84 => Prebid 1.15 => Prebid 1.15 w/ fix)_

• Brazil: 45% => 20% => 50%
• Chile: 45% => 25% => 45%
• China: 30% => 3% => 23%
• France: minor impact (-5%)
• Germany: minor impact (-5%)
• India: 25% => 8% => 25%
• Indonesia: 25% => 8% => 40%
• Malaysia: 40% => 8% => 45%
• Philippines: 30% => 5% => 30%
• United Kingdom: 45% => 35% => 55%
• United States: 48% => 22% => 43%

We realize only today that we were impacted in such proportion in the USA. In other western countries, the impact on the Prebid impression ratio was almost invisible. Surprisingly, we've even improved this ratio by 10% in the UK!
Anyway, for the so-called "exotic countries" you listed @Deimos01, the above figures confirm we're now back to normal ;)

[mkendall07]

Hey all. Thanks for the information. I going to go ahead and close this issue out. If you are still seeing a problem please feel free to open a new issue but we believe that most problems have been resoled. I've opened https://github.com/prebid/Prebid.js/issues/3004 to increase the default timeoutBuffer.

@Deimos01
I believe your issue is related to your implementation. Feel free to continue the conversation offline. Thanks again all!

[headerbidding]

Just so I understand it correctly: Is "timeoutBuffer: 400" the fix for the dropped impressions?

The reason I am asking is that I added "timeoutBuffer: 400" to my script on August 4 (using prebid 1.16) and impressions started to go up 13 days later (August 17).

(I am not sure if the server's cache control (10 days) + browser caching is responsible for the delay?)
(I also later on August 8 upgraded from prebid 1.16 to 1.19 )
I just like to better understand why impressions suddenly went up.

[mkendall07]

@headerbidding

Just so I understand it correctly: Is "timeoutBuffer: 400" the fix for the dropped impressions?

I believe it's a large contributing factor yes. I think also concurrency plays a role but to a lesser extend. It will depend on how many bidders you have and how you are configured etc.

Could definitely be caused by CDN caching. Hard to say exactly though without other data points.

[generic10one]

Can someone please explain what does 'timeoutBuffer' do exactly ?