Ax: [Question] reusing existing experiment data.

[showgood163]

The data I'm trying to reuse is from orthogonal search subspaces, e.g. the complete search consists of [h1,h2,h3,h4], and the data I want to reuse is the result of searching in [h1,h4] and [h2,h3].

I attached all the existing data (>50 for each subspace) to a GPEI model and used it to generate new trials. Here are my questions.

1. Is this way of reusing data encouraged? If not, what is needed to achieve this goal?
2. When generating new trials, sometimes I get errors like NaNs encounterd when trying to perform matrix-vector multiplication and sometimes it successed. I think it may be related to the random seed of the bayesian model, is that right? How can I avoid this?
3. Training and visualizing w/ the reduced hparam/metric set is OK. However, when visualizing w/ the full hparam/metric set using existing data, it _seems_ that I have to generate one trial with the bayesian model once. Is there any way to let the model be aware of the attached data w/o generating the next trial(which I think is memory consuming)?

Thanks!

[lena-kashtelyan]

Hi, @showgood163! As to your last question, you do need to fit the Gaussian Process model to all data once you add all the data, as I described in https://github.com/facebook/Ax/issues/202 –– you can achieve this either by generating the next trial or by separately fitting the model to the experiment extracted from the Service API as ax_client.experiment (refer back to the instructions in https://github.com/facebook/Ax/issues/202 if you prefer the latter path). The latter option should be less memory-intensive as the model is fit but does not need to generate the next trial.

cc @Balandat, @eytan, @bletham for questions 1) and 2).

[2timesjay]

@showgood163 Thanks for the interesting question. In general we hope to fit well when the whole search space is given at once. Have you tried this and received poor results, or alternately do you have a specific reason you're training the subspaces first?

More details about your use case might let us give a better alternative approach.

[showgood163]

@lena-kashtelyan
Thank you for pointing out that again! Yes, it works. I overlooked that.
I'd also like to point out that the script provided in the visualization tutorial needs to update, e.g. In [4]
model = Models.GPEI(experiment=exp, data=exp.eval())
should be
model = Models.GPEI(experiment=exp, data=exp.fetch_data())
or something like that.

@2timesjay
I'm searching for hparams for a NN model. These params can be divided into different parts, like architecture-related and regularization-related. I want to reuse the results that comes from subspaces after I realize that I may need to search on a larger parameter space.

I've tried this idea and here are the observations:

1. The trials generated from Model.GPEI are still within the subspaces w/ >60% probability.
2. When generating new trials, sometimes I get errors like NaNs encounterd when trying to perform matrix-vector multiplication, which I hardly encounter in the normal runs.

[lena-kashtelyan]

@showgood163, didn't mean to belabor what you already knew, was just answering the question of whether it's possible to let the model be aware of the attached data w/out generating more trials : )

Also, just to make sure –– you needed to do fetch_data with an Experiment (which you retrieved from AxClient, not a SimpleExperiment, correct? For SimpleExperiment, eval is the correct function.

[2timesjay]

@showgood163

If adding additional dimensions to your search space a single point on that dimension from past data doesn't allow the model to estimate a Gaussian Process lengthscale for that dimension (likely what leads to numerical errors).

The right thing to do here would be to do a small re-initialization of the entire new space and then re-adding the old data.

If extending a dimension that's already covers a range, then starting with the past data directly can work but initializing with a sobol search in the full new SearchSpace is always the most stable solution.