Abstract:

In the light of the conflict between limited memory and large-scale data, a solution to large volume concurrent data integration was proposed, which was not read the data into memory at one time, but divided the task into several sub-tasks according to the large volume data. The sub-tasks was complicated by the implementation so as to solve the limited memory and largescale conflict between the amount of data. The theoretical analysis and empirical experiments showed that resources could be used in the method of the system, and the solution could be efficiently applied in the large volume data integration process with a configurable mode.

Key words: data integration, large volume data, task decomposition