Gpyopt: Allowing constraints that have function calls

[iiternalfire]

The current implementation of "constraints" is very restricted as one cannot call a function defined in the workspace to be evaluated, thus general black-box constraints, output constraints, etc. cannot be realized in current GPyOpt. Consider the example below and see the concerns.

Minimal Code

from GPyOpt import Design_space, experiment_design

def fun(x): 
    return (6*x-2)**2*np.sin(12*x-4)

domain = [{'name': 'var_1', 'type': 'continuous', 'domain': (0,1)}]
constraint = [{'name': 'constr_1', 'constraint': 'fun(x) - 6.0'}]

feasible_region = Design_space(space = domain, constraints = constraint)
initial_design = experiment_design.initial_design('random', feasible_region, 10)

Error trace:

Fail to compile the constraint: {'name': 'constr_1', 'constraint': 'fun(x) - 6.0'}
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-41-d60bcb9c9d7d> in <module>
      2 constraint = [{'name': 'constr_1', 'constraint': 'fun(x) - 6.0'}]
      3 feasible_region = Design_space(space = domain, constraints = constraint)
----> 4 initial_design = experiment_design.initial_design('random', feasible_region, 10)

c:\users\212613144\appdata\local\continuum\anaconda3\envs\kchain\lib\site-packages\GPyOpt\experiment_design\__init__.py in initial_design(design_name, space, init_points_count)
     18         raise ValueError('Unknown design type: ' + design_name)
     19 
---> 20     return design.get_samples(init_points_count)

c:\users\212613144\appdata\local\continuum\anaconda3\envs\kchain\lib\site-packages\GPyOpt\experiment_design\random_design.py in get_samples(self, init_points_count)
     15     def get_samples(self, init_points_count):
     16         if self.space.has_constraints():
---> 17             return self.get_samples_with_constraints(init_points_count)
     18         else:
     19             return self.get_samples_without_constraints(init_points_count)

c:\users\212613144\appdata\local\continuum\anaconda3\envs\kchain\lib\site-packages\GPyOpt\experiment_design\random_design.py in get_samples_with_constraints(self, init_points_count)
     28         while samples.shape[0] < init_points_count:
     29             domain_samples = self.get_samples_without_constraints(init_points_count)
---> 30             valid_indices = (self.space.indicator_constraints(domain_samples) == 1).flatten()
     31             if sum(valid_indices) > 0:
     32                 valid_samples = domain_samples[valid_indices,:]

c:\users\212613144\appdata\local\continuum\anaconda3\envs\kchain\lib\site-packages\GPyOpt\core\task\space.py in indicator_constraints(self, x)
    305                 try:
    306                     exec('constraint = lambda x:' + d['constraint'], globals())
--> 307                     ind_x = (constraint(x)<0)*1
    308                     I_x *= ind_x.reshape(x.shape[0],1)
    309                 except:

c:\users\212613144\appdata\local\continuum\anaconda3\envs\kchain\lib\site-packages\GPyOpt\core\task\space.py in <lambda>(x)

NameError: name 'fun' is not defined

However, fun was defined just before being used to add an output constraint. In this case, one could have repeated fun script again in constraint, but this might not always be possible.

Concerns

1) Can you update exec('constraint = lambda x:' + d['constraint'], globals()) statement and constraint compiling codes to allow functions that are defined locally within the workspace?
2) Can you allow other blackbox constraints to be added just the way you allow external evaluation of objectives?

[ekalosak]

Thanks for the minimal code and stack trace.

I suspect this will not be satisfying, but this seems like a feature request that is motivated by a mis-specification of the optimization problem. In other words, you can indeed change the constraints dynamically if you insist by e.g. updating bo.context : ContextManager (GPyOpt/optimization/acquisition_optimizer.py has class ContextManager) - _but_ - that's an anti-pattern.

To move the issue forward, concretely, would you mind sharing the motivating problem for using fun(x)-6 in the domain?

[iiternalfire]

Hi @ekalosak , Sorry for delay in my response.

The concrete objective is to implement Bayesian optimization with unknown constraints [1], where analytical structure of the constraint is not known to us, but it can be evaluated point-wise by that function "fun(x)" in my example above. For example, if I am trying to do hyper-parameter optimization to get best accuracy under constraint that inference time on each instance is less that 60ms, then function fun uses timers to evaluate 95% upper confidence bound on inference time and then constraint is "fun(x) - 60 <= 0". See [1] for other examples.

[1] Gelbart, M.A., Snoek, J. and Adams, R.P., 2014. Bayesian optimization with unknown constraints. arXiv preprint arXiv:1403.5607.

[ekalosak]

Meta
This is clearly a valuable addition to GPyOpt - thanks for bringing it up. It's a substantial feature addition. If you bring up a draft branch, I'd bet some other contributors would help with some parts. Linus's Law in action >:)

Implementation
It's one thing to explicitly dis-allow portions of the design space. It's another to model the constraints as a conjugate Gaussian process in the acquisition function (Section 3.1 Eqn (7) in Gelbart et al. 2014) with arbitrary function calls. In this case, implementation-wise, it appears to be:

1. a new acquisition function (i.e. Eqn (7) in ibid)
2. an additional sidechain objective function return value (see evaluate_objective self.Y_new, cost_new, constraints_violated = self.objective.evaluate(self.suggested_sample) note the constraints_violated addition to the return signature) - or a list/dict of constraint functions should they be decoupled
3. additional models for the g_k(x) constraint functions (Section 2.2)
4. modifications to the main run_optimization loop to accommodate the previous additions e.g. how and when to evaluate g_k(x), when to optimize constraint surrogate model (g_k) hyperparameters, I'm sure there are other things I'm leaving out.

Do those requirements look right? As for an MVP (minimally viable product), the feature branch needs:

1. An external_constraints: Dict[str, Callable[List[...], Union[bool,float]]] = None optional kwarg in core/bo.py
2. An additional optional kwarg passthrough of external_constraints in the ModularBayesianOptimization class for top-level user-facing access.
3. A backwards-compatible implementation of the simplest method of constraint checking in the run_optimization function (i.e. do an if self.external_constraints is not None: self.check_external_constraints(...))

Notes
Following the evaluation stack down, it looks like a lot of this can go through the acquisition_optimizer's optimize() (here).

[sbatururimi]

Any update concerning this feature? Was also interested in.