Command line techniques

[or1k]

▪️ How to handle directories with a large number of files
You may have encountered this problem: you have tried to run ls on a directory with a very large number of files, but the command gives the error "argument list is too long". There are several ways to get around this limitation.

- Using the built-in command line
Built-in Bash does not have the ARG_MAX limitation.

- Use "find" when you need formatting parameters.

find /data/test_xargs -type f -ls -printf '%name'

Or with formatting that mimics ls:

find /data/test_xargs -type f -printf '%f\n

▪️ Suppose you want to compress all the files in a given directory from the previous example. 

You need a way to regulate the number of compression requests so that you don't run more processes than the number of processors you have.

Let's try to do this with find and xargs:

find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %

▪️ Match your processor to a set of tasks to maximize execution time
Despite CPU limits, some intensive tasks can slow down other processes on your computer as they search for resources. There are a few things you can do to keep server performance under control, such as using taskset.

The taskset command is used to set or get the CPU binding of a running process given its pid, or to start a new command with a given CPU binding. 

In general, we always want to leave one of the processors "free" for operating system tasks. Usually the kernel does a pretty good job of binding running processes to a specific processor to avoid context switching, but if you want to enforce which processors your process will run on, you can use taskset.

taskset -c 1,2,3,4,5,6,7 find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %