FileQueue

Bifrsot Diagram

Bifrost in the design framework, there is only one Binlog parser, and multiple table synchronization configurations for parallel synchronization In order to improve performance, Bifrost adopted channel queue cache and synchronized queue cache with two layers to realize parallel running of multiple table synchronization configuration.

Multiple tables run synchronously in parallel, which maximizes the synchronization performance. However, in the case of data conversion in a table synchronization or the target database fails to connect properly, the current table synchronization queue cache may appear, which indirectly affects binlog parsing blocking and other synchronization idle states.

Bifrost V1.2.1 , a new file queue feature was added to Bifrost. When the table synchronization blocked, the file queue would be started automatically, without affecting the Binlog parsing or other table data synchronization.

FileQueue Config

#Whether file queue is enabled. true|false
file_queue_usable=true

#Counts the time, in milliseconds, whether to start a file queue
file_queue_usable_count_time_diff=5000

#file_queue_usable_count_time_diff count of memory queue is crowded
file_queue_usable_count=10

file_queue_usable = false , file queues are not enabled, whatever the reason.

By default, file queues are written to memory queues 10 times in a row within 5000ms and start automatically, without any manual operation in the management interface.

File queue, is on table synchronization configuration for the unit start, that is, if a table is configured with 10 synchronization, synchronization is one of the first configuration synchronization to Redis, in this synchronous to the synchronization of Redis synchronization is slow, the memory queue blocked, has enabled a file queue, this table is used to buffer synchronization to the current synchronization to Redis