5

I have 60TB of data that resides in 12 csv files.

The data will be loaded into a clustered database where the loading processes is single threaded. In order to improve my load performance I need to initiate a load process from each node.

So far so good from this point of view. My biggest problem is how can I split this data? It is zipped, and each csv file has around 5TB of data! I tried split but it takes too long!

squillman
  • 38,163
Up_One
  • 150

3 Answers3

1

The easiest but not the fastest, most likely, way is

unzip -p <zipfile> | split -C <size>
Nik
  • 469
  • 3
  • 4
0

Assuming the order of the data unimportant, one way to do this -not so much of a faster way- but at least somewhat parallel would be to write a script that does the following.

  1. Open the zip file.
  2. Get the first file.
  3. Read the data out of the file, say in lines.
  4. For each csv line, write out a new zipfile containing the line.
  5. Rotate the file selected (say five zipfiles) using the output of one line.
  6. Once you reach a certain size (say 50GB) create a brand new zip file.

This isn't any faster than a sequential read of the big file, but allows you to split up the file into smaller chunks which can be loaded in parallel whilst the remaining data is completed.

Like most compressed output, its not seekable (you cannot jump X bytes ahead) so the biggest downside you have is if the process aborts for some reason you'd be forced to restart the whole thing from scratch.

Python provides support for doing something like this via the zipfile module.

Matthew Ife
  • 24,261
0

Do you have to load the 12 files in order or can they be imported in parallel?

I ask because it would seem that if they have to be loaded in order then splitting them further won't enable you to run anything in parallel anyway and if they don't then you can import the 12 files you already have in parallel.

If the files aren't already available on the nodes, transferring them there may take as long as the import anyway.

Bottlenecks can show up in surprising places. Have you started the single-thread import process and verified that the nodes are underutilised? You may be solving the wrong problem if you haven't checked.

Ladadadada
  • 27,207