High resources utilization
TDFS supports object storage and file storage structure, and supports most storage scenarios. More new features will be developed in the future to meet therequirements of different business scenarios, and to effectively perform resource flexibility and utilization.
TDFS is developed by Transwarp from the underlying architecture to the upper-level interface, and users have greater control.
Seamless integration of multiple components
TDFS combines the internal components of Transwarp to improve the efficiency of big data storage, providing storage capabilities for businesses with low-latency, high-throughput, and high-concurrency, and supporting "real-time" business scenarios.
Combined with self-developed Raft, when the new TDFS NameManager and BlockManager nodes are added to the cluster, their Raft nodes are automatically grouped, historical data is automatically synchronized, and retired nodes are automatically kicked out of the Raft group, thereby realizing non-aware cluster expansion and contraction.
TDFS abandons the QJM high-availability solution, it combines Raft to enable TDFS to re-select the master when the node is down, and continue to provide services, the switch between master and backup is at low cost. Its consensus mechanism ensures strong data consistency of each node and multi-replication partition tolerance. The BlockManager node of TDFS maintains the block information in the embedded database, and does not need to perform a full block report each time it is started, which effectively solves problems such as block report storms.
TDFS stores metadata in a lightweight embedded database and uses part of the memory cache, effectively solving the memory bottleneck problem caused by the storage of a large number of small files.
Different from the traditional HDFS + Zookeeper, as the core part of TDFS, the self-developed Raft does not need to start the service process, which reduces the O&M cost, makes TDFS more cohesive, and avoids the instability of external services.
Underlying language development
Based on the unique memory management mechanism of Rust language, TDFS does not require garbage collection, nor does it cause "stuttering" caused by GC. It provides compiler checking, which can effectively reduce various exceptions and concurrency problems, and does not require additional heap memory allocation. The memory is released when it is used up, which effectively reduces the cost.