Data Consumer And Ack
When a file queue is marked as started in a synchronous configuration and the current memory queue size is 0, data is automatically loaded from the file queue
Load process
Each time a data is read from the file queue, after loading out the write brush back to the memory queue,Until the memory queue is blocked.
Each time memory successfully synchronizes a piece of data, ack the data to the file queue.
After all the file queue data are processed, the mark file queue function is turned off, and the subsequent data is directly put into the memory queue.
Ack process
on ack,it’s not delete from the file queue immediately.
it is used to count how many pieces of data have been successfully processed for each file
on pop data from file queue, Statistics of each file data.
After the entire file has been loaded, mark that the current file has no data to load.
if unackCount = 0,the file will be deleted.
Pseudo code
type unackFileInfo struct {
id int64 // 文件编号
unackCount int // unack 数量
allInMemory bool // 全部数据已经加载到内存
totalCount int // 整个文件已经加载到内存消息条数
}
fileQueue = make(map[string]unackFileInfo,0)
fileQueue[fileName]=unackFileInfo{}
summary
Writing and reading are both orderly, which can ensure that the file is deleted, so the data has been processed successfully
Since this data is not deleted during ACK, if only half of the current file data is successfully processed and Bifrost is restarted, the next time the file is started, it will be processed from the first data of the current file.