Teammates: Admin create account: Cannot send Ajax Request!

[damithc]

I'm getting Cannot send Ajax Request! frequently when trying to create accounts, since two days ago. This almost never happened before.

[damithc]

This is serious. I can't create accounts at all. :-p

[damithc]

The reason is account creation is taking too long.

...
08:19:54.367
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.394
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.463
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.483
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.566
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.583
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.664
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.707
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:19:54.765
teammates.storage.api.EntitiesDb createEntities: Recently modified course::lburch.woo-demo2
08:20:17.014
/admin/adminInstructorAccountAdd
com.google.apphosting.api.ApiProxy$CancelledException: The API call datastore_v3.Get() was cancelled because the overall HTTP request deadline was reached while waiting for concurrent API calls.
    at com.google.apphosting.runtime.ApiProxyImpl$6.get(ApiProxyImpl.java:449)
    at com.google.apphosting.runtime.ApiProxyImpl$6.get(ApiProxyImpl.java:446)
    at com.google.appengine.tools.appstats.RecordingFuture.get(RecordingFuture.java:108)
    at com.google.appengine.tools.appstats.RecordingFuture.get(RecordingFuture.java:20)
    at com.google.appengine.tools.appstats.RecordingFuture.get(RecordingFuture.java:108)
    at com.google.appengine.tools.appstats.Recorder.processRecordingFuture(Recorder.java:397)
    at com.google.appengine.tools.appstats.Recorder.processAsyncRpc(Recorder.java:370)
    at com.google.appengine.tools.appstats.RecordingFuture.maybeRecordStats(RecordingFuture.java:140)
    at com.google.appengine.tools.appstats.RecordingFuture.get(RecordingFuture.java:110)
    at com.google.appengine.tools.appstats.RecordingFuture.get(RecordingFuture.java:20)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.getFutureWithOptionalTimeout(AsyncDatastoreServiceImpl.java:292)
    at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.aggregate(AsyncDatastoreServiceImpl.java:264)
    at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.get(AsyncDatastoreServiceImpl.java:235)
    at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.get(AsyncDatastoreServiceImpl.java:224)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:76)
    at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:63)
    at com.google.appengine.api.datastore.DatastoreServiceImpl.get(DatastoreServiceImpl.java:41)
    at org.datanucleus.store.appengine.RuntimeExceptionWrappingDatastoreService.get(RuntimeExceptionWrappingDatastoreService.java:56)
    at org.datanucleus.store.appengine.DatastorePersistenceHandler.get(DatastorePersistenceHandler.java:100)
    at org.datanucleus.store.appengine.DatastorePersistenceHandler.get(DatastorePersistenceHandler.java:112)
    at org.datanucleus.store.appengine.DatastorePersistenceHandler.fetchObject(DatastorePersistenceHandler.java:517)
    at org.datanucleus.state.JDOStateManagerImpl.validate(JDOStateManagerImpl.java:4263)
    at org.datanucleus.ObjectManagerImpl.findObject(ObjectManagerImpl.java:2444)
    at org.datanucleus.jdo.JDOPersistenceManager.getObjectById(JDOPersistenceManager.java:1671)
    at org.datanucleus.jdo.JDOPersistenceManager.getObjectById(JDOPersistenceManager.java:1767)
    at org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManager.getObjectById(DatastoreJDOPersistenceManager.java:73)
    at teammates.storage.api.AccountsDb.getAccountEntity(AccountsDb.java:211)
    at teammates.storage.api.AccountsDb.getAccount(AccountsDb.java:90)
    at teammates.storage.api.AccountsDb.getAccount(AccountsDb.java:101)
    at teammates.logic.core.AccountsLogic.isAccountAnInstructor(AccountsLogic.java:72)
    at teammates.logic.api.GateKeeper.isInstructor(GateKeeper.java:339)
    at teammates.logic.api.GateKeeper.getCurrentUser(GateKeeper.java:60)
    at teammates.ui.controller.ControllerServlet.doPost(ControllerServlet.java:130)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at teammates.storage.datastore.DatastoreFilter.doFilter(DatastoreFilter.java:29)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:50)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:260)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:78)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:148)
    at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:504)
    at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:446)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:453)
    at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:276)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:312)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:304)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:450)
    at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:235)
    at java.lang.Thread.run(Thread.java:745)

[damithc]

Use task queues to create the sample course?

[damithc]

@wkurniawan07 please get someone to work on this soon.

[wkurniawan07]

@JosephineKwa @chowyb @YongJieYongJie can we all (including me) try to work this out on our own? Whoever wants to submit a PR can feel free to do so, although Travis is not really being helpful at around this time.

[damithc]

Most likely we can optimize the sample course creation (reduce database calls).

[damithc]

Thanks for taking this issue @ablyx
The issue is no longer urgent because I've found a workaround ; I now use a separate server instance to create new accounts so that the operation is not slowed down by other requests being served by the same server instance. With that workaround, most of the time the operation succeeds although it still takes nearly the full 60 seconds, which is not good, and also adds to the cost. So it is good if we can fix this anyway.

[ablyx]

My pr merely shifts the logic.putDocument to a task queue. I think @whipermr5 has a better idea of putting all the documents at one go instead of doing it one by one. That might be a better improvement.