IT Brief Asia logo
Technology news for Asia's largest enterprises
Partner content
Story image

Bayesian optimisation: What it is, and how it can improve modelling

By Contributor
Tue 27 Jul 2021

Article by SigOpt AI product manager Meghana Ravikumar and research engineers Harvey Cheng and Michael McCourt.

Bayesian optimisation democratises access to scale, efficiency, and performance. Popularised initially as a way to break free from the grid, Bayesian optimisation efficiently uncovers the global maxima of a black-box function in a defined parameter space. 

In the context of hyperparameter optimisation, this black-box function can be the objective function: accuracy value for a validation or test set, loss value for a training or validation set, entropy gained or lost, AUC for ROC curves, A/B test performance, computation cost per epoch, model size, reward amount for reinforcement learning, etc.

There are a variety of attributes of Bayesian optimisation that distinguish it from other optimisation methods. In particular, Bayesian optimisation is the method that:

  • Explores and exploits a given parameter space to find the global optima
  • Robustly handles noisy data
  • Naturally adapts to discrete and irregular parameter domains
  • Efficiently scales with the hyperparameter domain.

Bayesian optimisation finds the global optima relatively quickly, works well in noisy or irregular hyperparameter spaces, and efficiently explores large parameter domains. Due to these properties, the optimisation technique is beneficial for hyperparameter tuning and architecture search of machine learning models.

The basics of Bayesian optimisation 

Step 1: Sample the parameter space

Initialise the process by sampling the hyperparameter space either randomly or low-discrepancy sequencing and getting these observations.

Step 2: Build a surrogate model

Build a probabilistic model (surrogate model) to approximate the true function based on given hyperparameter values and their associated output values (observations). In this case, fit a Gaussian process to the observed data from Step 1. Use the mean from the Gaussian process as the function most likely to model the black box function.

Step 3: Figure out where to sample next

Use the maximal location of the acquisition function to figure out where to sample next in the hyperparameter space. Acquisition functions play with the trade-off of exploiting a known high-performing result and exploring uncertain locations in the hyperparameter space. Different acquisition functions take different approaches to defining exploration and exploitation. 

Step 4: Sample the parameter space at the points picked on Step 3

Get an observation of the black box function given the newly sampled hyperparameter points. Add observations to the set of observed data. 

This process (Steps 2-4) repeats until a maximum number of iterations is met. Thus, by iterating through the method explained above, Bayesian optimisation effectively searches the hyperparameter space while homing in on the global optima.

Choose the right metric or metrics to optimise

Choosing the right metric or metrics is an essential step, as these values will be minimised or maximised by Bayesian optimisation. Doing this well can ensure that a model's performance aligns with end goals, facilitates fairness, or takes data properties into consideration. 

Optimisation algorithms will only amplify the chosen metric, so it's important to make sure these metrics reflect the organisation's goals. When in doubt about which metrics best reflect these goals, it's recommended to run some short optimisation cycles to better understand the hyperparameter space. 

Tracking and storing multiple metrics throughout the modelling and optimisation process will help organisations understand which metrics best relate to improved performance.

Integrate optimisation throughout your workflow 

Beyond hyperparameter tuning, Bayesian optimisation can help with data augmentation, feature engineering, model compression, neural architecture search, and much more. 

Taking optimisation into account earlier in a modelling workflow can help solve some of these problems. Furthermore, considering optimisation upfront will help alleviate engineering costs for parameterising models further down the line. 

In practice, we've seen significant improvements in performance and modelling workflow benefits when using Bayesian optimisation across a wide variety of models and problems, including:

  • Regression models: beat Wall Street by tuning trading models
  • Reinforcement learning: create a better agent for the classic cart-pole problem with Bayesian Optimisation
  • Data augmentation: use Bayesian Optimisation to augment your dataset
  • Deep learning architecture: tune model architecture and training parameters to quickly tune a CNN for sentiment analysis
  • Model compression: tune model distillation to achieve substantial model compression without loss in performance
  • Fine-tuning for image classification: identify the best-suited transfer learning technique for your problem
  • Unsupervised learning: tune a feedback loop to intelligently feature engineer.

Use a package that makes it easy to get up and running

When choosing the right Bayesian optimisation package for you, consider the following questions:

  • How much effort would it be to parameterise your existing code to integrate the package?
  • Is the package kept up to date?
  • Does the package offer features to make your optimisation cycles more efficient?
  • How easy is it to orchestrate and execute the package on your compute environment? 
  • Will you have to take care of parallelising Bayesian optimisation yourself, or is it built in?

Between open source and commercial offerings, there are plenty of Bayesian optimisation packages you could use. One of the most important considerations when making a selection is not related to marginal performance differences between them, but instead how easy they are to get up and running with your project. 

A fully supported package should include an easy way to integrate the optimisation loop in your code, recent releases that suggest it is maintained, automatic scheduling of next model configuration suggestions, and support for asynchronous parallelisation. 

Depending on your experimentation needs, you may also want to evaluate which features the package includes. For example, does it support all parameter types you need to optimise? Does it include multi-metric or multi-objective optimisation? Can you run multitask optimisation to use partial cost tasks to reduce the cost of your tuning job? There are many features that make a Bayesian optimisation package more or less useful for your particular modelling project that need to be considered.

What's next?

SigOpt has built an ensemble of optimisers that help organisations easily scale tuning, track experimentation, and organise modelling processes. With this API, organisations can power modelling with the experiment management solution and advanced HPO techniques such as Metric Strategy, Multimetric Optimisation, and easily scalable parallelism. 

To learn more about SigOpt, click here.

Related stories
Top stories
Story image
Delinea’s Joseph Carson recognised with OnCon Icon Award
Delinea chief security scientist and advisory CISO Joseph Carson has been recognised as a Top 50 Information Security Professional in the 2022 OnCon Icon Awards.
Story image
Artificial Intelligence
Dynatrace extends automatic release validation capabilities
Dynatrace has extended its platform release validation capabilities to improve user experience at every stage of the software development lifecycle.
Story image
Artificial Intelligence
Eight top DevSecOps trends to support IT innovation in 2022
The use of DevSecOps practices is growing, as it is increasingly seen as the best way to produce high-quality and secure code. So what are the current trends?
Story image
Multi Cloud
Cloud is a tool, not a destination
For many years, “cloud” has been thought of as a destination which has led to a misguided strategy that sees an enterprise trying to shift all its applications to a single cloud provider – regardless of the specific needs and nuances of each individual workload.
Story image
Zscaler launches co-located data centres in Canberra and Auckland
The investment will offer public and private sector enterprises greater resilience in support of their zero trust cybersecurity posture.
Story image
Zero trust security adoption rises 27% in just two years
A survey of WAN managers has revealed that multi-factor authentication and single sign-on are the top zero trust features implemented.
Story image
How to achieve your monthly recurring revenue goals
Monthly recurring revenue (MRR) is the ultimate goal, the most important issue on which anyone in the IT channel should focus.
Story image
Robust digital warehouse management crucial in Asia-Pacific
Thanks to a network of “cloud” stores, grocery and food delivery providers such as Foodpanda can arrange for these commonly requested items to get packed up and sent over in almost no time.
Story image
Internet of Things
ManageEngine wins big in IDC MarketScape assessment
ManageEngine's Endpoint Central service has been recognised as a leader by IDC MarketScape in several categories including Internet of Things device deployments and UEM software for SMEs.
Story image
New VMware offerings improve cloud infrastructure management
VMware has unveiled VMware vSphere+ and VMware vSAN+ to help organisations bring benefits of the cloud to existing on-prem infrastructure.
Find out how you and your business can prevent being caught out by everything from ransomware to cryptojacking.
Link image
Story image
NEC expands Open RAN ecosystem with Aspire Tech acquisition
With its agreement to acquire Aspire Technology, NEC Corporation has further increased its capacity to deliver End-to-End Open RAN ecosystems.
Story image
Preparing for the digital decade with the right workforce strategies
For a decade that started under the pall of the pandemic, the 2020s is poised to end with a bang with the digital economy swelling to a high across the world.
Story image
SNP unveils next generation of CrystalBridge software platform
Data is a key pillar of every customer-centric organisation, as it relies on agile decisions to become increasingly sustainable and intelligent.
Story image
How Airwallex helps businesses achieve globalisation success
As markets continue to shift, businesses need to be able to provide the same quality of service for customers regardless of where they are located around the world.
Story image
SAS wins Microsoft ISV 2022 Partner of the Year award
"We formed the SAS and Microsoft strategic partnership with a shared goal of making it easier for customers to drive better decisions in the cloud."
Story image
Voice recognition
Renesas and Cyberon expand services with voice recognition
“We are honoured to collaborate with Renesas to simplify the development of embedded voice recognition functions."
Story image
Oracle Cloud Infrastructure expands distributed cloud services
“Distributed cloud is the next evolution of cloud computing, and provides customers with more flexibility and control in how they deploy cloud resources."
Story image
Motorola Solutions
Motorola Solutions deploys communication system to 5th Japanese airport
Motorola Solutions with its partner, Nippon Airport Radio Services deployed the mission critical communication system to Kansai.
Story image
Evonik relies on Getac F110 tablet to control autonomous robot
The aim of the project is to evaluate the practicality of an automated robotic maintenance and inspection solution in the chemical industry.
Story image
Digital Transformation
What CISOs think about cyber security, visibility and cloud
Seeking to uncover the minds of CISOs and CIOs across Asia Pacific, my company recently asked Frost & Sullivan to take a snapshot of cloud adoption behaviour in the region.
Story image
Hybrid workforce
Why hybrid working is here to stay and how to ace it
Citrix's new report reveals hybrid workers are more productive and engaged at work than their office and completely remote counterparts.
Story image
Tech and data’s role in the changing face of compliance
Accenture's study found that 93% of respondents agree or strongly agree new technologies such as AI and cloud make compliance easier.
Story image
Supply chain
Supply chains continue to be disrupted, enterprises embrace circular economy
“Businesses urgently need to find a solution that can help them to manage this disruption, and transition to a circular economy."
Story image
FIDO Alliance releases guidelines for optimising UX with FIDO Security Keys
The new guidelines aim to accelerate multi-factor authentication deployment and adoption with FIDO security keys.
Story image
ASI Solutions named finalist of Microsoft Surface Partner of the Year
"ASI Solutions has a strong Microsoft focus, building value by helping customers maximise investment in modern workplace solutions."
Story image
Forescout reveals top vulnerabilities impacting OT vendors
Forescout’s Vedere Labs has disclosed OT: ICEFALL, naming 56 vulnerabilities affecting devices from 10 operational technology vendors.
Story image
Cloud Security
Palo Alto Networks bolsters cloud native security offerings
Latest Prisma Cloud platform updates help organisations continuously monitor and secure web applications with maximum flexibility.
Story image
Ivanti puts spotlight on power of employee digital experiences
The report revealed that 49% of employees are frustrated by the tech and tools their organisation provides and 64% believe this impacts morale.
Story image
Martech experts reveal the “buzz” on personalisation
In the digital age, innovative technology must be leveraged to power an efficient and effective relationship marketing strategy.
Story image
Vulnerable APIs costing businesses billions every year
Large companies are particularly vulnerable to the security risks associated with exposed or unprotected APIs as they accelerate digital transformation.  
Story image
NOWPayments launches new service to analyse cryptocurrency fees
NOWPayments has launched a new network fee optimisation solution that analyses current network fees and picks the most profitable option out of the client's payout wallets.
Story image
Industry-first comprehensive risk-based API security enhances protection
Application Programming Interfaces (APIs) have become a crucial part of operating web and mobile application businesses and are causing significant economic growth in the digital sector.
Story image
Mergers and Acquisitions
SAS acquires Kamakura to propel risk technology innovation
Underscoring SAS growth in the domain-specific solutions space, the acquisition will enable SAS to greatly enhance the breadth of its risk solutions portfolio. 
Story image
Stock security features inadequate in face of rising risk
"Organisations must proactively find ways of identifying unseen vulnerabilities and should take a diligent, holistic approach to cybersecurity."
Story image
Artificial Intelligence
Juniper study reveals top AI trends in APAC region
Juniper's research shows an increase in enterprise artificial intelligence adoption over the last 12 months is yielding tangible benefits to organisations.
Story image
Identity and Access Management
Ping Identity named a Leader in Access Management
Ping Identity has been named a leader in the 2022 KuppingerCole Leadership Compass report for Access Management. 
Story image
Video: 10 Minute IT Jams - An update from CrowdStrike
Scott Jarkoff joins us today to discuss current trends in the cyber threat landscape, and the reporting work CrowdStrike is doing to prevent further cyber harm.
Story image
Data ownership
Brands must reclaim trust by empowering data ownership
According to Twilio's new State of Personalisation Report 2022, 62% of consumers expect personalisation from brands, and yet only 40% trust brands to use their data responsibly and keep it safe.
Story image
New study reveals 51% of employees using unauthorised apps
The research shows that 92% of employees and managers in large enterprises want full control over applications, but they don't have it.
Story image
Robotic Process Automation / RPA
Salesforce announces latest generation of MuleSoft
Salesforce has introduced the next generation of MuleSoft, a unified solution for automation, integration and APIs to automate any workflow.
Story image
Artificial Intelligence
Accenture shares the benefits of supply chain visibility
It's clear that gaining better visibility into the supply chain will help organisations avoid excess costs, inefficiencies, and complexity to ultimately improve their bottom line.
Story image
SentinelOne integrates with Torq to empower security teams
"With Torq, security teams can extend the power of SentinelOne to systems across the organisation to benefit from a proactive security posture.”
Story image
Email threats spike 101%, remains a top attack vector
"Each year we see innovation in the threat landscape, but each year email remains a major threat to organisations."