Finding the minimum of a multimodal function

Level: Intermediate

Topic: Optimization

Can you find the global minimum of this multimodal function? Where in the design space does this occur and what is the corresponding function value? How would you explore this design space?

Please don’t look at the function definition and use your math smarts to know where the minimum is. Heck, if you can do that, that’s impressive, but that’s not really the point of this exercise.

import openmdao.api as om
import numpy as np


prob = om.Problem()

comp = om.ExecComp('obj = 0.1*(x+y) -abs(sin(x) * cos(y) * exp(abs(1 - (x**2 + y**2)**0.5/pi)))')

prob.model.add_subsystem('blackbox', comp, promotes=['*'])

prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'

prob.model.add_design_var('x', lower=-10, upper=10)
prob.model.add_design_var('y', lower=-10, upper=10)
prob.model.add_objective('obj')

prob.setup()

prob.run_driver();
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.6115904097239777
            Iterations: 7
            Function evaluations: 7
            Gradient evaluations: 7
Optimization Complete
-----------------------------------

Solution approaches

You have a few different options here. You could:

  • Simply plot the design space since this is a 2D function that’s relatively cheap.

  • Run a global gradient-free optimizer on the function.

  • Run a gradient-based optimizer from multiple starting points, keeping track of where the lowest value occurs.

Let’s walk through these together and I’ll discuss their pros and cons.

Plot the design space

This is the easiest and most intuitive way to see a design space. But this wouldn’t be reasonable in the general case for higher dimensional or expensive models. It also wouldn’t necessarily give you the optimum, but you could at least take a look and see how the function looks. Let’s do that now using the DOEDriver from OpenMDAO.

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline


prob = om.Problem()

prob.model.add_subsystem('blackbox', comp, promotes=['*'])

prob.driver = om.DOEDriver(om.FullFactorialGenerator(levels=41))

prob.model.add_design_var('x', lower=-10, upper=10)
prob.model.add_design_var('y', lower=-10, upper=10)
prob.model.add_objective('obj')
prob.driver.add_recorder(om.SqliteRecorder("cases.sql"))

prob.setup()
prob.run_driver();
/home/john/anaconda3/envs/course/lib/python3.8/site-packages/openmdao/recorders/sqlite_recorder.py:227: UserWarning:The existing case recorder file, cases.sql, is being overwritten.
/home/john/anaconda3/envs/course/lib/python3.8/site-packages/openmdao/visualization/opt_report/opt_report.py:123: DriverWarning:The optimizer report is not applicable for the DOEDriver Driver which does not support optimization
# check recorded cases from each case file
filename = "cases.sql"
cr = om.CaseReader(filename)
cases = cr.list_cases('driver', out_stream=None)

values = []
for case in cases:
    outputs = cr.get_case(case).outputs
    values.append((outputs['x'], outputs['y'], outputs['obj']))
values = np.squeeze(np.array(values))

figure = plt.figure()
contour = plt.tricontourf(values[:, 0], values[:, 1], values[:, 2], levels=21)
plt.colorbar()
plt.show()
../../../_images/finding_the_minimum_4_0.png

Oh wow, that’s a pretty rough design space. There are definitely a lot of local minima. But, based on this visualization we can sort of see four main low points near the corners. But which one of those four points is lowest? We probably need to dig deeper into the data, beyond just visualizing it.

sorted_values = values[values[:, 2].argsort(), :]
print(sorted_values[:3, :])
[[ -8.          -9.5        -20.66199056]
 [ -8.         -10.         -19.79686168]
 [  8.          -9.5        -19.06199056]]

All right, it looks like based on the plot data, the lowest point in the function space is -20.66199 at (-8, -9.5). But is that actually the lowest point in this space? Probably not because we had to discretize the space to plot it; we only have access to points on this grid. Let’s try some other methods.

Run a global gradient-free optimizer

We’ll now use a simple genetic algorithm from OpenMDAO to optimize this model.

prob = om.Problem()

prob.model.add_subsystem('blackbox', comp, promotes=['*'])

prob.driver = om.SimpleGADriver()
prob.driver.options['bits'] = {'x': 8, 'y': 8}
prob.driver.options['max_gen'] = 5

prob.model.add_design_var('x', lower=-10, upper=10)
prob.model.add_design_var('y', lower=-10, upper=10)
prob.model.add_objective('obj')
prob.driver.add_recorder(om.SqliteRecorder("GA_cases.sql"))

prob.setup()
prob.run_driver();
/home/john/anaconda3/envs/course/lib/python3.8/site-packages/openmdao/recorders/sqlite_recorder.py:227: UserWarning:The existing case recorder file, GA_cases.sql, is being overwritten.
# check recorded cases from each case file
filename = "GA_cases.sql"
cr = om.CaseReader(filename)
cases = cr.list_cases('driver', out_stream=None)

values = []
for case in cases:
    outputs = cr.get_case(case).outputs
    values.append((outputs['x'], outputs['y'], outputs['obj']))
values = np.squeeze(np.array(values))

sorted_values = values[values[:, 2].argsort(), :]
print(sorted_values[0, :])
[ -8.03921569  -9.76470588 -20.88419693]

Run multiple gradient-based optimizations

prob = om.Problem()

prob.model.add_subsystem('blackbox', comp, promotes=['*'])

prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'

prob.model.add_design_var('x', lower=-10, upper=10)
prob.model.add_design_var('y', lower=-10, upper=10)
prob.model.add_objective('obj')

np.random.seed(1234)
n_samples = 100

values = []
random_starts = np.random.rand(n_samples, 2) * 20 - 10
for (x, y) in random_starts:
    prob.setup()
    prob.set_val('x', x)
    prob.set_val('y', y)
    prob.run_driver();

    values.append((prob['x'], prob['y'], prob['obj']))
values = np.squeeze(np.array(values))
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.5163226799777205
            Iterations: 7
            Function evaluations: 9
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.548639620822383
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.185370448272815
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.551420299976121
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -8.047972489021983
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.864212749912046
            Iterations: 7
            Function evaluations: 8
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.5219023495075503
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -11.555259952709907
            Iterations: 8
            Function evaluations: 11
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -19.36996687742009
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.571121924446147
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.21898685990108
            Iterations: 7
            Function evaluations: 9
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.853988271458861
            Iterations: 4
            Function evaluations: 7
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.573863167636631
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.548639487893976
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.36101597957613
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.896564505995247
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.877690424901129
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.5219020415486604
            Iterations: 7
            Function evaluations: 7
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -7.285741216253686
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -7.285741190072775
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.963385516840605
            Iterations: 5
            Function evaluations: 9
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.202161732007603
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.611590273359305
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -5.5289861241353915
            Iterations: 8
            Function evaluations: 9
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.240703403002165
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.6675165522796
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -8.906913536945543
            Iterations: 3
            Function evaluations: 7
            Gradient evaluations: 3
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.961103710460614
            Iterations: 7
            Function evaluations: 9
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.614818715952591
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -17.437037218096012
            Iterations: 8
            Function evaluations: 13
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.5163226777953946
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.865086698310149
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -8.90691353707806
            Iterations: 5
            Function evaluations: 9
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -7.285741171579576
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.963385496358304
            Iterations: 6
            Function evaluations: 11
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.5219022520603236
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.231526297469605
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.585608935895351
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.8642130058468651
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.9789753719340295
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.0211842009289727
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.573863229459882
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.877690424932257
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.571121824904056
            Iterations: 4
            Function evaluations: 6
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.6115899917649823
            Iterations: 5
            Function evaluations: 9
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -5.528986147389784
            Iterations: 7
            Function evaluations: 8
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.614818686597976
            Iterations: 6
            Function evaluations: 10
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.8642129238343554
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.21898696512568
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.202161699701514
            Iterations: 5
            Function evaluations: 9
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.5219023260237914
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.2366544010053995
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.343900008204736
            Iterations: 6
            Function evaluations: 11
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.202161663214796
            Iterations: 8
            Function evaluations: 11
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -19.04805362701734
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.361016315225042
            Iterations: 6
            Function evaluations: 6
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -7.285741216401198
            Iterations: 8
            Function evaluations: 10
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -19.36996675397779
            Iterations: 5
            Function evaluations: 9
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.865086767434049
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.573863231639694
            Iterations: 5
            Function evaluations: 9
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.6148187162357
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.5219021032829243
            Iterations: 4
            Function evaluations: 6
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -5.85783926573642
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.516322678763987
            Iterations: 4
            Function evaluations: 7
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -20.98095921221227
            Iterations: 7
            Function evaluations: 10
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.667516811437427
            Iterations: 8
            Function evaluations: 11
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -5.225772319010425
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.36101630727762
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -8.361016315148248
            Iterations: 5
            Function evaluations: 5
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -10.361016233779605
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -16.30390683583646
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.8285837766790425
            Iterations: 3
            Function evaluations: 4
            Gradient evaluations: 3
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.853988328662048
            Iterations: 4
            Function evaluations: 7
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.8776903702930925
            Iterations: 4
            Function evaluations: 7
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.3624195900087614
            Iterations: 6
            Function evaluations: 6
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.3311030745719545
            Iterations: 3
            Function evaluations: 4
            Gradient evaluations: 3
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -5.857839314119491
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.5711218816188146
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -9.614818694416781
            Iterations: 8
            Function evaluations: 10
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -16.232351770570354
            Iterations: 11
            Function evaluations: 11
            Gradient evaluations: 11
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.5163226798988685
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -8.047972457265182
            Iterations: 7
            Function evaluations: 12
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -7.070195324579004
            Iterations: 5
            Function evaluations: 5
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.6115904352943982
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -17.437037247074016
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.3624195818978928
            Iterations: 4
            Function evaluations: 5
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.551420299306278
            Iterations: 8
            Function evaluations: 9
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.516322604197141
            Iterations: 7
            Function evaluations: 8
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -2.5486395852310393
            Iterations: 5
            Function evaluations: 7
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.5219022241621536
            Iterations: 6
            Function evaluations: 7
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.6115904383777737
            Iterations: 4
            Function evaluations: 6
            Gradient evaluations: 4
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -4.57386292929036
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -12.52601054221016
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.8642129645798633
            Iterations: 8
            Function evaluations: 9
            Gradient evaluations: 8
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -1.704275824225047
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -3.535572671379102
            Iterations: 7
            Function evaluations: 8
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -19.04805354510481
            Iterations: 7
            Function evaluations: 11
            Gradient evaluations: 7
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.853988334456597
            Iterations: 5
            Function evaluations: 8
            Gradient evaluations: 5
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -8.906913536963868
            Iterations: 6
            Function evaluations: 8
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
Optimization terminated successfully    (Exit mode 0)
            Current function value: -6.96110371762455
            Iterations: 6
            Function evaluations: 9
            Gradient evaluations: 6
Optimization Complete
-----------------------------------
sorted_values = values[values[:, 2].argsort(), :]
print(sorted_values[:3, :])
[[ -8.06001968  -9.66948136 -20.98095921]
 [  8.0499079   -9.6696239  -19.36996688]
 [  8.04990598  -9.66949297 -19.36996675]]

Cool! We’ve seen the lowest value yet, occurring at (-8.06002, -9.66948) and yielding -20.98096. Additionally, at least on my computer, this collection of gradient-based optimizations runs the quickest of any of the methods we’ve tried here. Obviously that’s problem-dependent, but it’s usually a good idea to start with multiple gradient-based optimizations instead of using a global gradient-free optimizer, just from a computational expense standpoint.

Recap

I hope this exercise proved to be useful for you to see how to investigate a pretty multimodal and sharp design space. Something like this could crop up in wind turbine layout design, aircraft engine performance optimization, or a slew of other cases. It really pays to systematically explore the design space and better understand it so that you can achieve optimal designs more quickly.