Category Archives: MySQL

Pipe mysqldump to ssh/sftp

If you need to send mysqldump over ssh on different server without creating any files on the local server, mostly if you are low on space you can use this:

mysqldump -u MYSQL_USERNAME -p DATABASE | gzip -c | ssh USER@HOST 'cat > ~/dump.sql.gz'


And for the restore from remote location:
ssh USER@HOST "cat /path/to/db.sql" | mysql -uUSER  -pPASSWORD DATABASE

Enabling query cache for MariaDB/MySQL

Query cache can speed up to 2-3 times queries that are often run.
It is pretty easy to set up. First you need to check if query cache is supported for your system:

show variables like 'have_query_cache';

That should return:

+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| have_query_cache | YES   |
+------------------+-------+

Then you need to check some variables related to query caching. I will explain each of them:

mysql> show variables like 'query_cache_%' ;
+------------------------------+----------+
| Variable_name                | Value    |
+------------------------------+----------+
| query_cache_limit            | 262144   |
| query_cache_min_res_unit     | 4096     |
| query_cache_size             | 10485760 |
| query_cache_type             | ON       |
| query_cache_wlock_invalidate | OFF      |
+------------------------------+----------+

query_cache_limit – the total size each individual query can be. In my case that is 262144
query_cache_min_res_unit – how large is each chunk/block of cached data
query_cache_size – the total amount of cached queries, 0 disables it, and it needs to be at least 40kb at the lowest to work. Usually the default is 16,777,216
query_cache_type – if the value is ON it means cache query is enabled. If it is OFF it means it is disabled.

Add the following to your sql config file, probably in /etc/mysql/my.cnf and edit the values to fit your needs and/or your servers specs –

[mysqld]
query_cache_type=1
query_cache_size = 32M
query_cache_limit=512K

Then restart mysql/maridb and you should be good to go. You can verify the changes in the config file by running again:

show variables like 'query_cache_%';

You can check some query cache stats by running this:

mysql> SHOW STATUS LIKE 'Qcache%';
+-------------------------+----------+
| Variable_name           | Value    |
+-------------------------+----------+
| Qcache_free_blocks      | 2        |
| Qcache_free_memory      | 14713664 |
| Qcache_hits             | 1750     |
| Qcache_inserts          | 1643     |
| Qcache_lowmem_prunes    | 0        |
| Qcache_not_cached       | 247      |
| Qcache_queries_in_cache | 423      |
| Qcache_total_blocks     | 887      |
+-------------------------+----------+

For further and more detailed information, MariaDB knowledge base about Query Cache is far better place.

MariaDB master-slave cluster on Ubuntu

This article explains how to run MariaDB SQL server in as master/slave replication cluster on two Ubuntu virtual machines.

master: 192.168.122.25
slave: 192.168.122.26

1. Before anything else you need to update all packages on the two machines:

sudo apt update
sudo apt upgrade

2. First thing is to add the official MariaDB repo for the stable release from here – https://downloads.mariadb.org/mariadb/repositories/
In my case, for Ubuntu 18.04 I had to use this:

sudo apt-get install software-properties-common
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ams2.mirrors.digitalocean.com/mariadb/repo/10.3/ubuntu bionic main'

3. Install it on both servers:

sudo apt install mariadb-server

You will have to provide password for the root user during install. Please note this is not the exisitng Ubuntu root user, but is new password the root user for mysql.

4. On both servers: sudo mysql_secure_installation
This will ask you for the root password you have set up in the previous step. You should remove anonymous users, disable remote root loginand remove test database. Basically answer yes[Y] to all if you are installing this on a machine available from the internet.

5. On both servers:

sudo systemctl enable mariadb.service
sudo systemctl start mariadb

The first command will make the mariadb server start every time the machine is re/started and the second will just the start service right now as it still not running.

6. On the master server create empty database

MariaDB [(none)]> mysql -uroot -p
MariaDB [(none)]> create database database_name;

7. On the master server we need to enable binary logging.
– Backup the original file in /etc/mysql/

cp my.cnf my.cnf.bkp

Add this new lines under the [mysqld] section, and replace the IP address with the one your master machine have.

#Replication settings
log-bin
server_id=1
bind-address=192.168.122.25
binlog-ignore-db = information_schema
binlog-ignore-db = mysql
binlog-ignore-db = performance_schema
binlog-ignore-db = test

This will replicate all new databases to the slave server, if you like to replicate just one specific database you should use

replicate-do-db = 

8. Now we need to login to the master sql server and create replication user and give the necessary grants.

MariaDB [(none)]> CREATE USER 'slave'@'localhost' IDENTIFIED BY 'SomePassword';
MariaDB [(none)]> GRANT REPLICATION SLAVE ON *.* TO slave IDENTIFIED BY 'SomePassword' WITH GRANT OPTION;
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> FLUSH TABLES WITH READ LOCK;
MariaDB [(none)]> SHOW MASTER STATUS;

The last command output is important in order the slave to know from which point it should start replicating from.

Unlock the databases and exit:

MariaDB [(none)]> UNLOCK TABLES;
MariaDB [(none)]> exit;

9. Login to the slave and create another empty database with the same name and the slave user.

CREATE DATABASE DATABASE_NAME;
CREATE USER 'slave'@'localhost' IDENTIFIED BY 'SomePassword';
FLUSH PRIVILEGES;

10. Add this the to the [mysqld] section in /etc/mysql/my.cnf in the slave:

server_id=2

Note that the master have server_id=1 so you should have different IDs on the different servers.

11. Log in to the slave database and run the following commands in the MariaDB prompt. Double check the MASTER_LOG_FILE and MASTER_LOG_POS variables, which should be the same as the values returned by SHOW MASTER STATUS above.

MariaDB [(none)]> CHANGE MASTER TO
MASTER_HOST='192.168.122.25',
MASTER_USER='slave',
MASTER_PASSWORD='SomePassword',
MASTER_PORT=3306,
MASTER_LOG_FILE='mariadb-bin.000001',
MASTER_LOG_POS=314,
MASTER_CONNECT_RETRY=10,
MASTER_USE_GTID=current_pos;

Now start the slave and check the status without exiting the MariaDB prompt:

MariaDB [(none)]> START SLAVE;
MariaDB [(none)]> SHOW SLAVE STATUS\G;

12. Test the replication:
login in the master server and create table in our empty database:

CREATE TABLE IF NOT EXISTS names (
task_id INT AUTO_INCREMENT,
title VARCHAR(255) NOT NULL,
start_date DATE,
due_date DATE,
status TINYINT NOT NULL,
priority TINYINT NOT NULL,
description TEXT,
PRIMARY KEY (task_id)
) ENGINE=INNODB;

You should see the new table created on the slave server too.

13. Debug: If there is something wrong with the slave replication it should show with when you run

SHOW SLAVE STATUS\G;

Most of the time problems are easily resolved with updating the slave configuration with the

CHANGE MASTER

query, stopping and then starting the slave. Watch for log position and the log file name.

MySQL tuning for low memory servers

If you are running couple of sites on a small VPS and you have 512-1024MB RAM, you are probably not amazed by mysql eating 400 just to start.
You can cut 200-300 of that memory usage by disabling performance schema in your config file – usually called my.cnf

performance_schema = off

If you are wondering if you should disable it and what will happen – you most probably don’t need that feature so it is safe to disable it. In a few words – it is to help you tune the SQL server, queries, find bottlenecks etc.
You can get more info here – MySQL performance schema

No space left on device.

Sometimes we can be fooled by error messages. For example one sunny day you see that for some reason your web or mail server doesn’t work. So you go to check the logs and find something similar to this:

2016/12/28 09:02:37 [crit] 24668#24668: *472674 open() "/var/cache/nginx/client_temp/0020878597" failed (28: No space left on device), client: 192.168.1.1, server: www.domain.com, request: "GET /cart/add/uenc/aHR0cDovL3d3dy5hYmNob21lLmNvbS9zaG9wL2xvdi1vcmdhbmljLWxvdi1pcy1iZWF1dGlmdWwtdGVh/product/19471/form_key/N8l3OyVkC1el9T8q/?product=19471&related_product=&send_to_friend=%2F%2Fwww.domain.com%2Fshop%2Fsendfriend%2Fproduct%2Fsend%2Fid%2F19471%2F&form_key=N8l3OyVkC1el9T8q&super_group%5B19425%5D=1&super_group%5B19424%5D= HTTP/1.1", host: "www.domain.com", referrer: "http://www.domain.com/shop/organic-tea"

Then when you check the free space you see that you have more than enough, and all kind of irrational thoughts start flowing into your mind, when it is the simple inodes space.

Usually it is just that there is not enough inodes left free on your files system, simple as that, but is easy to overlook as for some people this doesn’t happen often (and it shouldn’t).

[root@hostname client_temp]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/os-root 1703936 1703103 833 100% /
tmpfs 1524264 4 1524260 1% /dev/shm
/dev/sda1 51000 50 50950 1% /boot
/dev/mapper/os-tmp 131072 2155 128917 2% /tmp
/dev/mapper/data-data
19660800 578302 19082498 3% /data

Ubuntu 16.04 – Install Apache2 and php7

I have experimented some time ago, with php7, as described in this post. At that time that wasn’t official realease, however the link seems to pick up on some searches in google, and there was some confusion for people expecting this to be copy/paste guide.

So I would try to fix these and outline the steps to install Apache2 with php7 on the latest server LTS Ubuntu – 16.04.01 – freshly downloaded from the official site, at the time of the writing.

First, the two most repeated commands:


sudo apt-get update

sudo apt-get upgrade

That will update any software installed on the server distro.

Then install apache2 –

sudo apt-get install apache2

Open the IP of the server you have installed it on and you should be presented with the default apache web page, if nothing is showing, you need to check if apache2 is running and if the firewall/ufw is blocking requests.

It comes the turn to install mysql –

sudo apt-get install mysql-server

You will be asked to provide password for the MySQL root user, you should be aware this is not the same user as the Linux root user, it is different one having rights to do everything with every database in MySQL, so it is a good idea to pick a different password then the ones you are using currently in the system.

After installation is finished, we will have to run buil-in script to tighten some of the security for MySQL and clean up some things –

sudo mysql_secure_installation

You will be asked for you the root password (MySQL root user), and then to choose a level for password validation:


There are three levels of password validation policy:

LOW Length >= 8

MEDIUM Length >= 8, numeric, mixed case, and special characters

STRONG Length >= 8, numeric, mixed case, special characters and dictionary file Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 1

They are self explainatory, but I don’t really like password validators – I find them stupid security measure, especially on the level of MySQL server/users. So I would set that to 0/low for machines I am using.

You should read the questions and answer then with ‘y’ or ‘no’, in the past it was fine to answer all with ‘y/yes’ however, now I am noticing that the first question is “Change the password for root” – you might not really want to do that, so the best thing is to read what are you actually asked.

 

Ok, so we are getting there, let’s install php7 with the apache mod –

sudo apt-get install php7.0 libapache2-mod-php7.0 php7.0-mcrypt php7.0-mysql

Then you could check the php version in the terminal with –

php -v

And the final step is to be sure apache is interpreting php in the browser.

First, become root with –

sudo -i

Then will add php info page to the server web root directory, so we could open it in our browser after that to verify it is running properly on check all the configuration details for php –

echo '<?php phpinfo(); ?> > /var/www/html/info.php'

After that you should navigate in your browser to the IP your server is listening to, and add /info.php after it, so it would look something like this –

http://192.168.122.113/info.php

That’s pretty much for it, but this where the complicated things starts from.

Dump and compress all MySQL databases

This is a little variation of a script I used in the past, what will do is find all databases, and export them compressed, with piping to gzip.

#!/bin/bash
MYSQLDUMP="$(which mysqldump)"
DIR="/root/db_dumps"
DBS=`mysql --defaults-extra-file=.my.cnf -e"show databases"`

#Remove	old dumps
rm -f $DIR/*.sql*

for DATABASE in $DBS
do
if [ $DATABASE != "Database" ]; then
FILENAME=$DATABASE
$MYSQLDUMP --defaults-extra-file=.my.cnf $DATABASE --single-transaction | gzip > $DIR/$FILENAME.sql.gz
fi
done

rm -f $DIR/information_schema.sql*
rm -f $DIR/performance_schema.sql*

.my.cnf file contents:

[client]
user=root
password=your_password

.my.cnf file should be in the user directory owned by him with permissions 600 or even 400