Feed: MemSQL Blog.
Author: Floyd Smith.
Gaming has gone from a niche activity to an entire industry focused on delivering real-time digital experiences. Gameloft, an award-winning developer and publisher of mobile and console games, has used MemSQL to extract actionable insights in real time from more than 100TB of raw data.
According to Juniper Research, the gaming industry will touch $132 billion in total revenue by 2021 – well ahead of Hollywood. It is a fiercely competitive business, where every game developer’s dream is to shoot to the top of the charts.
Gameloft, as a French-based leading publisher and developer of mobile and console games, has built a devout fanbase, spread all across the globe. All games are developed in-house, and they have emerged as one of the top innovators in the industry.
Their in-house franchises boast winning titles such as Asphalt 9: Legends (download) and Dragon Mania Legends (download), and their top-tier collaborators include entertainment brands known around the world. Gameloft has won many awards; Minion Rush and Asphalt 8: Airborne are featured in the “Top 10 iOS Games by All-Time Worldwide Downloads,” from App Annie. Asphalt 9: Legends won a design award from Apple and two Webby Awards. You will find them on lists of the top gaming companies in the world.
The rising popularity of mobile games has driven Gameloft to create unique experiences for all digital platforms – gaming consoles and PCs, of course, but also smartphones, mobile phones, tablets (including Apple iOS and Android devices), set-top boxes, and connected TVs. With its global distribution network spanning 150-plus countries, and many millions of daily active users across the globe, Gameloft aims to marry the virtual and real-world through online gaming, every single day.
But, with all this success, Gameloft had a business need for outstanding performance, in order to deliver real-time gaming KPIs, and to conduct business with customers – including in-game transactions – in real time. To do so, they had bet on a robust data infrastructure platform that will meet their current and future gaming needs. As shown by their many successes, and their close collaboration with leading players, Gameloft’s is running a serious business, and the stakes are high.
Without a strong, ready-to-go data infrastructure to power their games, all the hard work done to create a compelling game could quickly go wasted. Gameloft experienced issues with:
- Processing the high influx of real-time key performance indicator (KPI) data across their growing fanbase.
- Effectively capitalizing on critical micro-moments in the game to deliver the right experiences, at the right time, and to make the right offers to customers.
Gameloft was stuck behind delays in their analytics pipeline.
No Real-Time Insights
Today, nearly every industry is data-driven and needs actionable data. Gameloft was no different, and without real-time data insights, Gameloft could not fully optimize its monetization of games.
For Gameloft, real-time insights were needed to drive several in-game tactics, including feature promotion, selective campaigning, and player retention schemes. With analytics delayed, they didn’t know how much revenue the game was making at a given point in time, and how they could keep the game more enjoyable for the players, keeping the game “sticky.” Without this, they were unable to capitalize on many important factors, like promoting the right features at the right time, or running a campaign based on player demographics and in-game behavior. There was a need to offer, for instance, free game coins at just the right moment, so a player can purchase more lives and continue playing.
Gameloft Runs into Scaling Challenges
Gaming is a “hits” business, and when a game goes viral, it’s crucial to be able to scale the backend infrastructure instantly. Several Gameloft games involve characters from major entertainment brands. These games are designed using the same spirit and atmosphere as in hit Hollywood movies. It was clear that the growth potential for these games was unlimited.
When the need comes up for one of these games to scale, it needs to scale immediately. Some games that experience a successful first few days quickly lose momentum due to performance and availability issues. There have been games that hit a peak, as new users flood in – but then experience responsiveness issues, and get pulled from the market a few days later.
Gameloft didn’t want to meet the same fate, and the legacy MySQL backend at Gameloft was not meeting their expectations: the need to calculate in-game KPIs, in real time, from live interactions with millions of users. Throw a few more million concurrent users into the pool of live accounts, and the MySQL database would run into speed and concurrency limitations that slowed down the experience for everyone.
Gameloft clearly had to offload some of the work from the MySQL backend to scale. MemSQL’s elastic and robust technology turned the odds in their favor. Let’s see how…
Old Architecture using MySQL
Before they moved to MemSQL, Gameloft began by storing their KPIs in a MySQL cluster running on commodity hardware. This system was used for both vital KPIs that had stringent SLAs (such as the number of active users, player retention data, and user pings), and less-critical KPIs with relaxed SLAs. The MySQL cluster comprised 16 servers, with data manually partitioned across the different servers. In addition, the application servers had to be aware of which data resided where.
Before the data hit the MySQL cluster, Gameloft used an off-the-shelf technology called MooseFS to stage incoming KPI data locally on their servers. MooseFS is a fault-tolerant, highly available, scale-out distributed file system. ETL jobs then processed the data from MooseFS, reading and transforming it before it was stored in the MemSQL or the MySQL cluster. Metrics related to vital KPIs were sent to MemSQL, and the others were sent to MySQL.
In terms of data size, the MySQL database had grown to roughly 100TB of raw data and counting, and Gameloft was not able to extract actionable insights in real-time. Gameloft was using stored procedures that took nearly 12 hours to generate the insights needed to improve the gaming experience. At that point, the value of the insights diminished – because the user’s in-game experience was already further along, and it would be out of context to influence the game with stale insight data. (Imagine driving your car, and getting a low fuel alert when the car is already out of gas.)
Re-engineering for High-Speed Analytics and Scale with MemSQL
As a solution to its real-time insights issues, Gameloft decided to adopt MemSQL, which offers high performance on ingest and queries, and a widely used SQL interface for queries and analytics applications. To reduce the risk initially, they opted for a Lambda-like architecture, where fast KPIs were processed using MemSQL, and the less critical KPIs were still processed using MySQL.
MemSQL is a fully scalable relational database that supports SQL – but in a distributed fashion, such that MemSQL can offer scalability that’s unusual in relational databases. With inherent SQL support, MemSQL can be used as a nearly drop-in replacement for most existing relational databases, including MySQL. The built-in MySQL wire protocol support in MemSQL means that it works “out of the box” without any application rewrites. This is in stark contrast to NoSQL databases, which are heavily used in the gaming industry and are scalable, but lack SQL support. This makes them both difficult to write queries for and slow to respond to queries.
For many teams at Gameloft, using MemSQL meant zero impact and no code changes, but tremendous benefits, from both the architectural and application points of view. So, how was MemSQL able to deliver the blazing fast KPIs needed by Gameloft? Well, the answer lies in the fact that MemSQL is a unique technology that marries in-memory with disk-based database technology and also provides a column store engine.
With MemSQL at the heart of Gameloft’s new architecture, the fast KPIs were calculated at high speed providing near real-time data. From immediately analyzing KPI data, to using real-time intelligence to capitalize on certain game features, campaigns and timed events, Gameloft would never miss an important in-game moment. After moving from MySQL to MemSQL, Gameloft is seeing “Analytics complete within 15 minutes, which used to previously take 12 hours,” says Marian Nicolae, Business Intelligence Manager at Gameloft.
In the case of Gameloft, MemSQL was used for a specific use-case of speeding up KPI calculations, in combination with MySQL. What makes MemSQL stand out, compared to those other tools, is performance. With over 1.5TB of data in MemSQL, Gameloft’s deployment of MemSQL still had plenty of headroom for growth. Additionally, MemSQL at Gameloft was deployed on premise, but in the future Gameloft could easily migrate their MemSQL database to the cloud (with Helios), or use a hybrid solution to further reduce costs.
Getting Started with MemSQL
If you have workloads that require fast ingest, fast processing of large data tables, and fast queries, in operational analytics or other use cases, try MemSQL for free today, or contact us to learn how we can help you.