Category Archives: Linux

Mount SSHFS volumes in fstab with ssh key

SSHFS on command line usually takes the ssh key with the -o, option which doesn’t really work when adding it in fstab. If you look around the internet you most probably found that people recommend adding fstab records with the -o option, but this wont work, simply add the key as another regular fstab option –

sshfs#USER@domain.com:/data/www /mnt/logs/  fuse IdentityFile=/home/USER/.ssh/id_rsa,uid=UID,gid=GUID,users,idmap=user,noatime,allow_other,_netdev,reconnect,ro 0 0 

Replace USER with the user who connects to the remote server and UID and GUID with the ones from the remote server.
The above also mounts the remote system as read-only so you wont be able to write on the mount.

No space left on device.

Sometimes we can be fooled by error messages. For example one sunny day you see that for some reason your web or mail server doesn’t work. So you go to check the logs and find something similar to this:

2016/12/28 09:02:37 [crit] 24668#24668: *472674 open() "/var/cache/nginx/client_temp/0020878597" failed (28: No space left on device), client: 192.168.1.1, server: www.domain.com, request: "GET /cart/add/uenc/aHR0cDovL3d3dy5hYmNob21lLmNvbS9zaG9wL2xvdi1vcmdhbmljLWxvdi1pcy1iZWF1dGlmdWwtdGVh/product/19471/form_key/N8l3OyVkC1el9T8q/?product=19471&related_product=&send_to_friend=%2F%2Fwww.domain.com%2Fshop%2Fsendfriend%2Fproduct%2Fsend%2Fid%2F19471%2F&form_key=N8l3OyVkC1el9T8q&super_group%5B19425%5D=1&super_group%5B19424%5D= HTTP/1.1", host: "www.domain.com", referrer: "http://www.domain.com/shop/organic-tea"

Then when you check the free space you see that you have more than enough, and all kind of irrational thoughts start flowing into your mind, when it is the simple inodes space.

Usually it is just that there is not enough inodes left free on your files system, simple as that, but is easy to overlook as for some people this doesn’t happen often (and it shouldn’t).

[root@hostname client_temp]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/os-root 1703936 1703103 833 100% /
tmpfs 1524264 4 1524260 1% /dev/shm
/dev/sda1 51000 50 50950 1% /boot
/dev/mapper/os-tmp 131072 2155 128917 2% /tmp
/dev/mapper/data-data
19660800 578302 19082498 3% /data

Nginx basic authentication

Setting up basic authentication in Nginx is pretty easy, you need to first add couple of directives in block config, you can out them directly in the server if you want the whole site not be accessible or just on some parts-

    auth_basic "Restricted Content";
    auth_basic_user_file /etc/nginx/.htpasswd;

Then in the .htpasswd file add the user and use openssl to generate hash for the chosen password –

echo "password" | openssl passwd -apr1 -stdin

Encrypt and decrypt files with openssl

If you need easy way to encrypt and decrypt files on linux systems, one way you can go is to use openssl, usually it is available on most servers,

Encyrpt:

openssl aes-256-cbc -e -in file.zip -out file.enc -k password

Decrypt:


openssl aes-256-cbc -d -in file.enc -out file.zip -k password

 

You should also be able to use it with des3, but I haven’t tested the fallowing:

Encrypt:

openssl des3 -salt -in file-plain.txt -out file-encrypted.txt.des3

Decrypt:

openssl des3 -d -salt -in file-encrypted.txt.des3 -out file-plain.txt

Calculate total size of files listed

Sometimes you need to know the combined size of a given files that are combined with other files in the same dir, either by their extension or by some other property.

This example is based on file extension, but can be adapted to any other results you like find to return:

find . -name "mysql-bin.*.1" -size +1000000c -print0 | du -hc --files0-from=- | awk 'END{print $1}'

Stop and remove all docker containers and images

If you want to save some time removing all containers, you can use this two command(the first one is to stop all if there are some running):

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

You can also remove all the images with:

docker rmi -f $(docker images -q)

Be aware that the -f option will force delete even images that are being used.

Hits per second, hits per minute

In case you need to generate file from a log with the number of hits per second or minute you can use the fallowing commands-

Hits per second –

cat access.log | cut -d[ -f2 | cut -d] -f1 | awk 'BEGIN { FS=":" } ; { print $1":"$2":"$3":"$4 }'| sort | uniq -c >/home/user/hits-per-sec

Hits per minute –

cat access.log | cut -d[ -f2 | cut -d] -f1 | awk 'BEGIN { FS=":" } ; { print $1":"$2":"$3 }' | sort | uniq -c >/home/user/hits-per-min

SFTP upload on one line

This can be used in a script to schedule uploads, however it is not recommended to use it and store passwords in plain text, but sometimes you might not have choice if you don’t control the remote server.

You will also need the sshpass package installed.

sshpass -p "password" sftp user@remote.com:/directory <<< $'put /var/www/www.local.com/httpdocs/var/export/*.Products.csv'