Ways to Split Large File for CSV or TXT on Linux Mint –The text based file contain many rows of data and you can split them into separated file easily on Linux. How you can split large text file into multiple files on Linux let’s follow these guide.
Split a large file into smaller files on Linux can be done with Terminal, the command promt on Linux and you can also Keep Header Row of the file.
For an example you have a CSV file which contain around 1000 rows; each row having 5 columns. You want to break this CSV file into 10 CSV files of 100 records each. (Each having same CSV header as present in original CSV). How to solve this case ?
This is the result of files that I have tried using these method:
Ways to Split Large File for CSV or TXT on Linux Mint
Linux has a great little utility called split, which can take a file and split it into chunks of whatever size you want, eg 100 line chunks.
However, for CSV files etc, each chunk generally needs to have the header row in there. Unfortunately the split command doesn’t have an option for that. However with a little bit more code you can achieve it.
If yuo have CSV file called hotel.csv which contain 1000 rows; each row having 5 columns. You want to break this CSV file into 10 CSV files of 100 records each.
Place that file into your home directory and open Terminal (press Ctrl + Alt + T), then type the following command :
tail -n +2 hotel.csv | split -l 100 – split_
Press enter then enter this command :
for file in split_*; do head -n 1 hotel.csv > tmp_file; cat $file >> tmp_file; mv -f tmp_file $file; done
Press enter again, for the result you can see the created new file named split_aa.csv, split_ab.csv, …etc on your home directory. Now you have finished split large text file into smaller files on Linux.
You can also execute the command one by one per line into Terminal as follow:
tail -n +2 hotel.csv | split -l 100 - split_
for file in split_*
head -n 1 hotel.csv > tmp_file
cat $file >> tmp_file
mv -f tmp_file $file