Feed: MemSQL Blog.
Author: Floyd Smith.
Deriving insights from your data is a competitive advantage that can no longer be ignored. But many companies are finding it difficult to make the most of their data, because traditional data technology wasn’t built with such huge scale in mind.
A traditional SQL-based database could handle almost any workload, but the workload had to fit on a single machine. If you needed to scale – to support real-time analytics, for instance – you had to buy into NoSQL solutions – a multitude of specialized components to tackle the rising demands on your data infrastructure.
Here’s a representative data architecture. You might be dealing with a similarly complex data infrastructure at your company today.
There’s a better way – and Fanatics can show all of us how one fast-growing, global, multi-billion dollar company did it.
How Fanatics Grew into Complexity – and Beyond
As the global leader in licensed sports merchandise, Fanatics is changing the way fans purchase their favorite team apparel and jerseys across retail channels through an innovative, tech-infused approach to making and selling fan gear in today’s on-demand culture.
Fanatics has its own supersite, Fanatics.com, but also runs the online stores of all major North American sports leagues, more than 200 professional and collegiate teams, and several of the world’s largest global football (soccer) franchises. Even as it grows rapidly, Fanatics is profitable, with revenues exceeding $2B a year.
Three years ago, Fanatics decided to completely revamp its technology architecture. It had to meet ever-growing user needs for responsiveness, reliability, and scalability as Fanatics added several leagues, teams, and franchises globally. The real-time nature of Fanatics’ innovative mobile and ecommerce platform allows it to react quickly to both planned and unplanned moments throughout the sports calendar, including Championships, player trades, and record-breaking performances, where fan demand for products quickly spikes.
Fanatics moved from siloed and monolithic applications to a completely event-driven architecture. At the core of event-driven architecture is a Kafka-based messaging backbone called Fanflow.
Fanflow uses Kafka to move events from online stores, mobile interactions, and point of sale systems through the payments backend and into the analytics system. Events include user behavior interaction on the site, type ahead suggestions, searches, product recommendations, etc. Events are also generated for all the state changes within an order life cycle, such as additions to cart, checkout, fulfillment, and delivery.
At the same time, Fanatics had intensive analytics needs – needs that were not being met by the old way of doing things.
Open Source Analytics
Analytics is crucial to Fanatics’ business. The company has to work with partners to anticipate, respond to, and fulfill customer ordering peaks and valleys. Team wins and losses can generate hard-to-fulfill surges, or a team’s loss or a player trade could result in merchandise markdowns.
As part of the move to Fanflow, Fanatics adopted an analytics approach that combined a few open source technologies running against cloud-based storage. Event information was stored in JSON, which caused difficulties when Fanatics later evaluated traditional SQL database solutions.
The company used Flink and Redis for state management as it computed business KPIs and time-series metrics in real time. It created Spark applications for in-depth analytics, such as subscription/order attribution, activity, and clickstream models. Fanatics used Lucene-based indexers for its primary persistence layer, and for data discovery and simplified dashboarding.
used different tools for different audiences
There was a split between the tools and functionality available to different audiences. Managers and executives used a Lucene-based index for queries and limited data discovery. Data scientists and data engineers used Hive and Zeppelin to analyze the same events at a deeper level.
This caused challenges. Fanatics’ workflows were very complex and difficult to manage during peak traffic events, such as a championship game. Business needs evolve frequently at Fanatics, meaning schemas had to change to match but these changes were very difficult to manage with the Lucene-based indexers.
Maintaining the different query platforms and the underlying analytics infrastructure cost Fanatics a lot of time to keep things running and meet SLAs. So the company decided on a new approach.
Re-Engineering Analytics with MemSQL
As a solution to its analytics issues, Fanatics decided to adopt MemSQL, which offers high performance on ingest and queries and a widely used SQL interface for queries and analytics applications. MemSQL combines in-memory and disk-based technology, including a columnstore engine. MemSQL also offers built-in interoperability with Apache Kafka and JSON, supporting previous Fanatics technology choices.
MemSQL replaced the Lucene-based indexers. Spark and Flink jobs were converted to SQL-based processing, which allows for consistent, predictable development life cycles and more predictability in service level agreements (SLAs).
The Fanatics event stream runs at an average of 2,500 events per second, or 4 million events per day, and the MemSQL database has grown to roughly 3 billion rows and counting. Yet users are still able to run ad hoc queries and scheduled reports, with excellent performance.
With the previous mix of tools, Fanatics was not able to do data integration across enterprise sources. With MemSQL, Fanatics regained that capability. The company ingests all its enterprise sources into MemSQL, integrates the data, and gains a comprehensive view of the current state of the business.
Fanatics was also able to unify its audiences onto a single, SQL-based platform. This sharply lowers the barriers to entry, as SQL is so widely known. Fanatics no longer has to set up different kinds of platforms for different audiences. The development team now spends a lot more time deriving deeper insights and developing new capabilities, such as self-service analytics platforms, washboarding, and others.
The team also supports an increasing number of use cases, such as order visibility, which powers several customer service applications internally. The teams are able to spend more time providing deeper insights from the data rather than just keeping a plethora of platforms up and running.
The Future for Fanatics
A tech company at heart, Fanatics is remaining ahead of the curve in today’s lightning fast, mobile economy through significant investments in technology, data, on-demand manufacturing and global infrastructure. The company has reinvented the way high-quality fan gear and apparel is quickly designed, manufactured and delivered through its innovative vertical-commerce (v-commerce) model, built to satisfy the increasingly real-time expectations of fans and retailers worldwide with a wider assortment of high-quality products.
Fanatics continues to grow rapidly, opening offices and manufacturing/distribution facilities in several of the world’s sports-crazed markets. Fanatics is backed by some of the world’s foremost investors, most recently having secured a $1B injection from SoftBank’s Vision Fund, the largest tech fund in history, which identified Fanatics as one the world’s transformative companies helping to shape the future of commerce. Executive chairman Michael Rubin has been named one of the Ten Influencers of 2018 by media site SportsPro.
As the company grows, so will Fanatics’ commitment to continually improving its application architecture and analytics capabilities.
Click to view last month’s Strata Data keynote, featuring MemSQL’s Drew Paroski and Aatif Din of Fanatics.