26
Traditional Deployment Framework Innovation and IT System Enhance the Efficiency Real-time and Streaming Data Processing Capabilities
For the past half century, the basic design philosophy of databases has been the “request sent + result return” model, which has also been borrowed by Apache Storm, Spark Streaming, and Flink stream processing frameworks. However, the stream processing framework focuses on “processing” and does not have the capabilities of a database itself. At that time, all “real-time data processing” capabilities were built outside the database engine, and the database that was most closely aligned with the data did not realize its full value. Before business flows need to be associated with other data or interact with temporary storage, developers must first write complex Java/C++/Scala code to complete data extraction, preprocess records one by one in the most traditional way, and frequently retrieve additional data from the outside for manual association, which brings great burden to development, operation, and maintenance work.
Therefore, by integrating the “streaming data processing” capability into the database and defining the processing logic and topology of real-time data through SQL+ triggers, the development difficulty and O&M complexity of real-time applications of the database can be greatly reduced. Based on this concept, AsiaInfo AntDB has become one of the few databases in China with the capability of “hyper-convergence + streaming real-time data warehouse”.Hyper-convergence Framework Creates a New Era of Distributed Databases
In the past decade, the rapid development of the financial and Internet industries with large data scale, query complexity, and high correlation has promoted the “distributed” and “cloudified” process of domestic databases. Distributed databases have smooth scaling and high reliability, and “cloudification” is conducive to reducing database O&M costs and enabling flexible resource scheduling.
In this decade, in the process of “digital and intelligent transformation” of hundreds of industries, users’ demand for databases has become increasingly refined, and multi-service support capabilities have been favored by enterprises. In this context, multi-engine databases have gradually developed fusion capabilities, and HTAP, lakehouse integration, and stream batch integration are the embodiment of this trend.
AsiaInfo’s AntDB team proposes a new concept of “hyperconvergence”, that is, the integration of multiple engines and capabilities to meet the increasingly complex business needs of enterprises with mixed workloads and mixed data types. The AntDB “hyperconverged” framework can make full use of the structural advantages of the distributed database engine, and unify the packaging of multiple engines such as time series storage, stream processing execution, and vectorized analysis on the basis of HTAP (online transaction and online analysis). The same database cluster can support multiple business models, meet diversified data requirements, greatly reduce the complexity of business systems, and realize “one-stop data management” under a unified framework.
Figure: AntDB database “hyperconvergence” framework
Streaming engine disrupt the database kernel that hasn't changed for 50 years
The streaming data processing pattern is very different from the kernel design of a traditional database. In the traditional database framework, the application and the database are in a “request-response” relationship, while the streaming kernel is a “subscription-notification” model, through a pre-defined data model, business “events” are processed, and then the database actively pushes the processing results to downstream applications for use, which is like completing mobile phone operations conveniently through Siri or Xiaoai (Xiaomi’s AI assistant) in the mobile phone.
Figure: AntDB database streaming engine infrastructure
At the end of 2022, AsiaInfo launched the AntDB-S stream processing database engine, which comprehensively and thoroughly integrates streaming computing with traditional transactional and analytical data storage, allowing users to freely define data structures and real-time processing logic through standard SQL in the database engine. At the same time, during the free flow of data between stream objects and table objects within the database, users can optimize the performance, process, cluster monitoring, and business logic customization of data at any time by establishing indexes, stream table associations, triggers, and materialized views.
Figure: AntDB streaming engine and functional benefits
AsiaInfo’s AntDB-S Streaming Database Helps Enterprises Real-time with the Whole Business Chain
AntDB-S streaming database is suitable for real-time data warehouses, real-time reports, real-time alarms, asynchronous transactions and other business scenarios. Users can use simple SQL to create complex streaming data processing business logic, easily replace Apache Storm, Spark Streaming, Flink and other traditional streaming engines, save a lot of development and testing resources, and fundamentally ensure data security and ACID characteristics (indivisibility, consistency, independence, durability). In addition, AntDB-S also has high availability, multi-tenancy, authentication, distribution, DBRA, transaction processing and other capabilities, which greatly reduces the user’s streaming service development and maintenance costs.
Figure: AntDB-S Streaming Database typical scenarios
In traditional framework, data needs to be extracted from the database for analysis; the technology stack is complex; the operation is slow, and the enterprise IT system is getting larger and heavier; the extraction process will incur high operation and maintenance costs. AntDB transforms the complex “data storage + data computing + data management” into a “hyper-converged” integrated framework, which can achieve faster kernel-level streaming analysis and processing through AntDB-S products, bringing instant insight, smarter decision-making, and faster application integration to enterprises with simpler technology and less cost, helping enterprises transform digitally.
