Partitioning a large directory into subdirectories by size

August 11, 2020

Since I’m not fond of carrying around all my photos on a cell phone where they’re perpetually at list of loss, I peridiocally dump the image and video files to a drive on my desktop for later burning to optical disc.¹ Saving these images in archival form is a hedge against the bet that my existing backup system won’t fail someday.

I’m using Blue-Ray optical discs to archive these image and video files; and each stores 25 GB of data. So my plan was to split the iPhone image dump into 24 GB partitions. H

#!/bin/bash      
                                                               
directory=${1:-testdir}                                                         
sizelimit=${2:-1000} # in MB                                                    
sizesofar=0                                                                     
dircount=1                                                                      
du -s -m "$directory"/* | while read -r size file                  
do                                                                              
	if ((sizesofar + size > sizelimit))                                           
	then                                                                          
		(( dircount++ ))                                                            
		sizesofar=0                                                                 
	fi                                                                            
	(( sizesofar += size ))                                                       
	mkdir -p -- "$directory/sub_$dircount"                                           
	mv -- "$file" "$directory/sub_$dircount"                                           
done

We call the script with ./dump-split.sh dir_to_split your_size_limit. In my case, the size limit (in MB) was 2400. This script is modified slightly from a version for Unix.² The du tool differs between these two platforms. This script doesn’t optimize the packing of the discs, but over the many files being archived, it all evens out.

See also