Split the file "file.txt" into files beginning with the name "new" each containing 20 lines of text each.
Type man split at the Unix prompt for more information. However you will have to first remove the header from file.txt (using the tail command, for example) and then add it back on to each of the split files.
Made it into a function. You can now call splitCsv <Filename> [chunkSize]
splitCsv() {
HEADER=$(head -1 $1)
if [ -n "$2" ]; then
CHUNK=$2
else
CHUNK=1000
fi
tail -n +2 $1 | split -l $CHUNK - $1_split_
for i in $1_split_*; do
sed -i -e "1i$HEADER" "$i"
done
}
This should do it for you - all your files will end up called Part1-Part500.
#!/bin/bash
FILENAME=10000.csv
HDR=$(head -1 $FILENAME) # Pick up CSV header line to apply to each file
split -l 20 $FILENAME xyz # Split the file into chunks of 20 lines each
n=1
for f in xyz* # Go through all newly created chunks
do
echo $HDR > Part${n} # Write out header to new file called "Part(n)"
cat $f >> Part${n} # Add in the 20 lines from the "split" command
rm $f # Remove temporary file
((n++)) # Increment name of output part
done
file_name = Name of the file you want to split. 10000 = Number of rows each split file would contain file_part_ = Prefix of split file name (file_part_0,file_part_1,file_part_2..etc goes on)
https://stackoverflow.com/a/53062251/401226 where the answer has comments about installing the correct version of parallel (in ubuntu use the specific parallel package, which is more recent than what is bundled in moreutils)
This question was asked many years ago, but for future readers I'd like to mention that the most convenient tool for this purpose is xsv from https://github.com/BurntSushi/xsv
The split sub-command is meant to do exactly what has been asked in the original question. The documentation says:
split - Split one CSV file into many CSV files of N chunks