Hopsworks

Hopsworks is an AI and data platform specialising in building, maintaining and operating AI and machine learning (ML) systems at scale. The company started life as an academic research project run by co-founder Dr. Jim Dowling during his tenure at leading Swedish university KTH. There he worked on HopsFS, a scalable distributed file system that was developed to address data bottlenecks in AI systems. HopsFS then evolved into Hopsworks, a data science platform with a unique core technology centered around a feature store. It became part of pan-European project HEAP (Human Exposome Assessment Platform) which provides data for research institutes, in this case, specifically helping to combat cancer. 

Hopsworks was formally established in 2017 and now has 40 staff around the world. The company provides an AI lakehouse built around its pioneering feature store—an abstraction layer that manages the complete lifecycle of ML features with built-in lineage, versioning and governance. By combining lakehouse capabilities with real-time feature serving through a unified API, Hopsworks significantly accelerates organisational AI projects including LLM creation, predictive analysis, logistics, KYC and image analysis.

“If you train and serve AI models in the same systems, you’ll sometimes get data loss and shift. This can result in models not making it to product stage. With our AI lakehouse, you can abstract the layers away while keeping the different stages logically separate, dramatically accelerating time-to-market for AI projects.”

Lex Avstreikh, Hopsworks Head of Strategy

Avstreikh and the team were keenly aware that although Hopsworks had been in the AI sector since 2017, things were ever-changing, and the future was uncertain.

“We wanted our platform—and the future of the company—to be as open to change as possible,” Avstreikh explained. “We knew that it was important to avoid using domain-specific languages (DSL) which would lock our users into certain architectural choices as these could end up being very poor bets in the future! We needed all of our stack, especially the infrastructure layer, to give us as much flexibility as possible.” 

The Hopsworks team tried a number of different cloud providers, but found many of them difficult to use, insufficiently sustainable, and unable to resolve their growing concern about data sovereignty. Furthermore, when it came to the new Hopsworks serverless platform—which is based on Kubernetes and provides free storage and model deployment to customers—it was important to keep costs down, and have an easy-to-upgrade infrastructure. Although the team could scale up to 20,000 users without additional costs, the potential egress costs with their current hosting provider presented a real financial risk. 

“Many of the cloud platforms have accumulated processes that I can best describe as obstructive technical debris,” continued Avstreikh. “Pricing could be unclear and the toolsets didn’t always work well together—it often felt like playing with building blocks in the dark. Once, we accidentally incurred a bill after leaving a storage service on by accident and we were keen to avoid this in the future.”  

As a result, Avstreikh and his team began to look for an alternative infrastructure provider that would be compatible with their existing toolset. It was also important that the cloud environment was easy to use, preferably local, and kept their options open for building the platform in the future. The team also wanted a platform provider whose ethos aligned with Hopsworks’ environmental commitment and philosophy of being as sustainable as possible. 

As a European company, Hopsworks’ team had come across OVHcloud before, and started a rigorous analysis of how it compared to their current cloud provider. OVHcloud’s services out-performed competitors’ in almost all of the areas that were important to Hopsworks—and at a lower cost.

Hopsworks started to migrate its serverless offering to OVHcloud’s Public Cloud environment, using its S3 compatible Object Storage solution. The team also joined OVHcloud’s startup program, benefiting from additional business and technical support. 

“We had a great experience with the OVHcloud startup program. The mentoring aspect was very good, and we could get access to people across the business, including commercial support when we needed it. It’s a very practical program, which we really appreciated.”

  Lex Avstreikh, Hopsworks Head of Strategy

The Hopsworks team backed up their cluster with their main provider before starting the migration to OVHcloud. Although there was a small, predictable amount of downtime, Hopsworks had notified users of the maintenance, and there were no user queries or issues with accounts as a result of the migration. 

The Hopsworks team tracked costs across the process, including running Managed Kubernetes, network egress, S3 storage, container registry, data transfers and EBS instances, and they reported a 62% saving on total cloud costs. 

“We started the process gradually, and although there are always minor hiccups with any migration, we’ve managed to make considerable savings by moving to OVHcloud whilst also ensuring that our service is hosted in a European Cloud,” Avstreikh said. “There haven’t been any crisis meetings, complaints, and no one has ever suggested going back to our previous provider! We also like that OVHcloud is sustainable by design, and that our cloud services are as efficient as possible.” 

“OVHcloud’s Managed Kubernetes service is also very straightforward to use, and I’ve tried almost all of the large cloud’s alternatives. Finally, thanks to the startup program, we have built really good relationships across the business: whatever my query, I can always find someone to help. All in all, we’re very happy with how things worked out.”

  Lex Avstreikh, Hopsworks Head of Strategy