Documentation / Tutorials / Server
Commonly needed Linux commands
/ Server / Commonly needed Linux commands
At WPoets we manage all our servers in house, so here we have listed some of the most common commands that we use.
Here is the command that gives us exact disk space.
This will give you the size of each folder within the folder path you have run the command
du -sh */ .*/ | sort -n
If you to count the directories in a folder you do so by
ls -lR | grep ^d | wc -l
and, if you need the list of files just do
ls -1R | wc -l
To count files (even files without an extension) recursively from the root of the current directory, use:
ls -lR | grep ^- | wc -l
Here is an alternative
tree share/some/directory/ | tail -1
Dealing with Zip and Tar
Creating a Tar.gz file from a folder
tar -zcf wordpoets_com.tar.gz /_data/htdocs/
where wordpoets_com.tar.gz is the file name and /_data/htdocs/ is the folder path we want to compress.
If you want to create tar file, with a relative path instead of full absolute url you can do something like this
tar -czf /var/www/backups/full/2019/20190807.tar.gz -C /var/www/backups/db/fullbackup/2019/07 20190807
It will create the tar file relative to 20190807 instead of /var/www/backups/db/fullbackup/2019/07/20190807 which is what it generally does.
Let's say the above folder you want to extract within /htdocs/ folder, without creating another /htdocs/ within, use the following command
tar --strip-components=1 -xzvf wordpoets_com.tar.gz
the --strip-components allows you to set the depth while extracting the tar file.
If you need to compress a file or uncompress it we can use the following command, we generally use it to compress SQL files
it will create the compressed version of the file as abc.sql.gz and delete the abc.sql, and if you want to uncompress the abc.sql.gz file just type the following command to get the original file back.
On servers managed by us, we use Fail2ban actively to block IPs and threats, but sometimes we end up banning IPs wrongly in those times from command easily allows you to check which IP was blocked by which rule, making it easier to identify
fail2ban-client status | sed -n 's/,//g;s/.*Jail list://p' | xargs -n1 fail2ban-client status
and, if just want to see list of jails active run following
If you know the name of specific jail, you also use the following command to check the list of banned IP addresses
fail2ban-client status wordpress2
where wordpress2 is the name of the jail specified in fail2ban jail.conf file.
Finally, to unban an IP, you can use the following command
fail2ban-client set wordpress2 unbanip 18.104.22.168
just remember to replace the wordpress2 and the IP address with your own.
Once in a while, we need to run wget in a cron, and one the trouble was ignoring the output here is the setting that does that
wget -q -O - https://www.wpoets.com/crons/modules/add-hub-note >/dev/null 2>&1
-O - option makes sure that the fetched content is sent to stdout.
another alternative is
wget -O /dev/null -o /dev/null https://www.mangalamjobs.com/site-notifications/candidate-reports
Here -O sends the downloaded file to /dev/null and -o logs to /dev/null instead of stderr. That way redirection is not needed at all.
Rsync & SCP
When moving files between two servers, we currently prefer either rsync or scp depending on the situation.
rsync -a --partial /var/www/wordpoets.com/ /var/www/dev.wordpoets.com/ #folders are matched… rsync -a --partial /var/www/dev.wordpoets.com/db [email protected]:/var/www/wordpoets.com/ #db folder will be created on new server nohup rsync -a --partial /var/www/sync-vayas [email protected]:/root/sync/& #this will create sync-vyas folder in remote sync folder nohup rsync -av --partial --rsync-path "sudo rsync" /var/www/dev.wordpoets.com/htdocs [email protected]:/var/www/ls1.wordpoets.com/& #to run sudo on remote rsync -rP --ignore-existing /var/www/delphianlogic.com/htdocs2/wp-content/uploads/ /var/www/delphianlogic.com/htdocs/wp-content/uploads #ignore existing files in destination
-r to recurse into directories,
--ignore-existing to ignore existing files in the destination,
-P if you want to watch progress,
--list-only if you want to see what it would copy without actually copying, and a
-t if you want to preserve the timestamps.
scp -vrC /var/www/inc/ [email protected]:/var/www/inc/ #Copy all the files form inc folder to remote inc folder scp foobar.txt [email protected]:/some/remote/directory #Copy the file "foobar.txt" from the local host to a remote host scp [email protected]:foobar.txt /some/local/directory #Copy the file "foobar.txt" from a remote host to the local host scp -r foo [email protected]:/some/remote/directory/bar #Copy the directory "foo" from the local host to a remote host's directory "bar"
–r Recursively copy entire directories. Note that this follows symbolic links encountered in the tree traversal.
-C Compression enable. Passes the -C flag to ssh to enable compression.
-l limit – Limits the used bandwidth, specified in Kbit/s.
-o ssh_option – Can be used to pass options to ssh in the format used in ssh_config.
-P port – Specifies the port to connect to on the remote host. Note that this option is written with a capital ‘P’.
-p Preserves modification times, access times, and modes from the original file.
-q Quiet mode: disables the progress meter as well as warning and diagnostic messages from ssh.
-v Verbose mode. Print debugging messages about progress. This is helpful in debugging connection, authentication, and configuration problems.