ThoughtSpot prioritizes the high availability and minimal downtime of our systems to ensure a seamless user experience. In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction. As the demand for high performance analytics continues to grow, our systems must be engineered to accommodate increasing data volumes while maintaining high speeds and reliability.
To tackle these challenges, we use an in-house metadata store called “Atlas”. Atlas keeps our system responsive even under heavy workloads. Optimizing the server initialization process for Atlas is vital for maintaining the high availability and performance of the ThoughtSpot system. This allows users to interact with their data without interruption, regardless of system scale.
This article highlights the performance optimizations implemented to initialize Atlas, our in-house Graph database, in less than two minutes. These enhancements ensure ThoughtSpot’s high availability and reliability, even as data volumes continue to scale.
What is metadata?
In ThoughtSpot, metadata is a collection of objects that define the system's structure and behavior. These objects include users, groups, data connections, tables, data models, search results, Liveboards, and so on. All these objects are essential for managing access, configuring data connections, and building interactive Liveboards. Nearly all user interactions involve accessing and modifying the metadata objects. Therefore, efficient storage, management, and retrieval of metadata are crucial for ThoughtSpot's overall performance.
What is Atlas?
Atlas is an in-memory, multi-versioned Graph database, implemented in Java to manage connected objects. It stores all the metadata created within a ThoughtSpot instance to enable efficient querying, retrieval, and management of data objects. While Atlas operates as an in-memory graph database for speed and performance, it uses PostgreSQL as its persistent storage layer to ensure durability and long-term data storage. This hybrid approach allows Atlas to benefit from the speed of in-memory operations while maintaining data persistence through Postgres.
Most user interactions involve accessing or modifying metadata, so Atlas must perform at high speeds to keep the system responsive and capable of handling real-time queries. This is critical to delivering a fast and interactive analytics experience to ThoughtSpot users.
Atlas stores the graph in PostgreSQL by flattening the objects and relationships into the following tables:
A table named storables, stores data objects (nodes and edges) with unique identifiers, versions, content, and deletion status. These data objects are referred to as “storables”.
A table named index, stores metadata about indexes associated with the schema, including name, version, and node type.
A table named commit, that stores information about schema commits, including commit number and time.
Atlas manages the relationships between objects and metadata in a graph structure, where nodes represent objects and edges represent relationships. This graph-based structure enables a highly scalable and efficient way to organize and traverse metadata. At system startup, Atlas must initialize the graph which involves loading metadata objects into memory and establishing nodes and edges. This process is essential for ThoughtSpot to operate with high availability, because it ensures that the graph database is ready for rapid querying and processing as the data scales.
Server initialization time
Embedded analytics implementations often support a large number of customer organizations, each of which may have a substantial number of users. Additionally, every customer organization can have its own unique metadata. This combination can result in extremely large volumes of metadata.The Atlas server encountered significant performance bottlenecks during initialization in environments with large volumes of metadata. For example, instances containing over 3 million objects and 8 million relationships required almost an hour to fully initialize. This prolonged initialization time resulted in extended downtimes during critical operations, such as instance upgrades or patch applications, and impacted ThoughtSpot's overall availability and user experience.
The goal was to reduce the server initialization time to less than 5 minutes and bring back the Atlas server online quickly, with minimal disruption to operations. This improvement was essential for maintaining ThoughtSpot's high availability and reliability in environments with massive data scales.
Optimizing server initialization time
To reduce the initialization time for Atlas, we focused on three core areas of optimization:
#1 Eliminating String Interning
One of the most impactful optimizations was removing string interning during initialization. String interning is a common optimization mechanism where duplicate strings are stored in a pool to save memory, but can result in performance issues.
Challenges with string interning
Hash Collisions: During initialization, we detected a large number of distinct UUIDs. These strings caused collisions in the string pool’s underlying hash table, leading to performance degradation.
Operational Overhead: Tuning the number of buckets with XX:StringTableSize in the string pool to handle such a high volume of strings is complex and not scalable for instances that keep growing.
Optimizations
Removing the interning of UUID strings during initialization eliminated hash collisions and improved overall computational performance. As distinct UUIDs were being assigned during the server initialization process, the need for interning was eliminated.
After eliminating string interning during initialization, we saw a marked reduction in latency. This optimization is particularly notable because string interning is a common performance technique used across the industry. However, it resulted in latency and performance issues when processing a high volume of unique strings for ThoughtSpot.
#2 Optimizing querying Postgres
One of the primary bottlenecks was the way metadata was retrieved from our persistent storage layer, Postgres. The existing queries were inefficient for large-scale metadata due to full table scans and batch processing.
Challenges
Full table scans fetched large amounts of data, including unnecessary metadata, slowing down the process.
Metadata was retrieved in small batches (1,000 at a time), leading to multiple iterations and redundant table scans.
Optimizations
We restructured the queries to use an inner join between the index and storable tables, eliminating the need for separate queries to filter storables. Fetching all relevant records from the storables table at once and bypassing unnecessary iterations significantly reduced the load on the database.
This step reduced the number of database queries and time to fetch metadata from Postgres, optimizing data retrieval.
#3 Enhancing in-memory processing
After retrieving metadata, we encountered another bottleneck during in-memory operations. The conversion of storables into Atlas nodes and edges was handled sequentially, which became inefficient when processing large volumes of metadata.
Challenges
Sequential processing of fetched data to build Atlas nodes and edges.
Sequential processes to build the header graph entities, such as typedNodes, InEdge, and outEdge adjacency lists
Limited utilization of the modern multi-core CPU architecture
Optimizations
We introduced parallel processing using Java's parallel streams and thread executors. By processing independent tasks concurrently, we parallelized the creation of nodes, edges, and other entities in the header graph and significantly improved the overall speed of the operation.
Results
Our optimizations resulted in significant improvements in the server initialization time.
In instances with 3 million objects and 8 million relations, it reduced from 54 minutes to 2 minutes and 24 seconds
In instances with 1.5 million objects and 5 million relations, it reduced from 42 minutes to 1 minute and 46 seconds
These reductions were achieved without making significant architectural changes to Atlas. Efficient data retrieval, parallel in-memory processing, and elimination of interning in string operations helped optimize the Atlas startup time and performance.
CPU and Memory Impact
CPU: The parallel processing approach resulted in higher CPU utilization during initialization. However, the server was dedicated to initialization during this time, so the system was able to handle the load efficiently.
Memory: While short-lived memory consumption increased, garbage collection handled it effectively, and memory usage normalized once initialization was complete.
Conclusion
By optimizing queries to Postgres, leveraging parallel processing, and eliminating string interning, we reduced Atlas initialization time from nearly an hour to just a few minutes.
Java offers nuanced functionalities such as intern() and StringTableSize, which can significantly impact system performance. While default configurations are often adequate, a deeper understanding of the language features and their underlying memory model can help developers optimize their application performance more effectively.
This article emphasizes the importance of targeted performance tuning in large-scale systems like ThoughtSpot. We expect that these optimizations will continue to benefit even when our metadata grows in volume and complexity.
