Cloud Hosting

The most mesmerizing way of optimizing arbitrary functionsPhoto by James Wainscoat on Unsplash

Whether we deal with machine learning, operations research, or other numerical fields, a common task we all have to do is optimizing functions. Depending on the field, some go-to methods emerged:

It is always great if we can apply these methods. However, for optimizing general functions so-called blackbox optimization we have to resort to other techniques. One that is particularly interesting is the so-called particle swarm optimization, and in this article, I will show you how it works and how to implement it.

Note that these algorithms wont always give you the best solution, as it is a highly stochastic and heuristic algorithm. Nevertheless, its a nice technique to have in your toolbox, and you should try it out when you have a difficult function to optimize!

In 1995, Kennedy and Eberhart introduced particle swarm optimization in their paper of the same name. The authors draw an analogy from sociobiology, suggesting that a collective movement, such as a flock of birds, allows each member to benefit from the experiences of the entire group. We will see what this means in a second.

Let us assume that you want to minimize a function in two variables, for example, f(x, y) = x + y. Sure, we know that the solution is (0, 0) with a value of 0, and our algorithm should find this out as well. If we cannot even do that, we know that we did something completely wrong.

We start by randomly initializing a lot of potential solutions, i.e., two-dimensional points (x, y) for i = 1, , N. The points are called particles (birds), and the set of points is the swarm (flock). For each particle (x, y), we can compute the function value f(x, y).

More here:
Particle Swarm Optimization. The most mesmerizing way of optimizing | by Dr. Robert Kbler | Jan, 2024 - Towards Data Science