Jump to content

Command line techniques

Recommended Posts

▪️ How to handle directories with a large number of files
You may have encountered this problem: you have tried to run ls on a directory with a very large number of files, but the command gives the error "argument list is too long". There are several ways to get around this limitation.

- Using the built-in command line
Built-in Bash does not have the ARG_MAX limitation.

- Use "find" when you need formatting parameters.

find /data/test_xargs -type f -ls -printf '%name'

Or with formatting that mimics ls:

find /data/test_xargs -type f -printf '%f\n

▪️ Suppose you want to compress all the files in a given directory from the previous example. 

You need a way to regulate the number of compression requests so that you don't run more processes than the number of processors you have.

Let's try to do this with find and xargs:

find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %

▪️ Match your processor to a set of tasks to maximize execution time
Despite CPU limits, some intensive tasks can slow down other processes on your computer as they search for resources. There are a few things you can do to keep server performance under control, such as using taskset.

The taskset command is used to set or get the CPU binding of a running process given its pid, or to start a new command with a given CPU binding. 

In general, we always want to leave one of the processors "free" for operating system tasks. Usually the kernel does a pretty good job of binding running processes to a specific processor to avoid context switching, but if you want to enforce which processors your process will run on, you can use taskset.

taskset -c 1,2,3,4,5,6,7 find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %
Link to comment
Share on other sites

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...

Important Information

By using this site you automatically agree to the Privacy Policy | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.