System Architecture for Scaling Virtual Environment

All traffic should have ssl certs installed for each domain at nginx level.  Nginx is configured as ssl proxy only.  Certs are provided by Let’s Encrypt unless otherwise needed for other purposes.

(non-secure) Varnish (80) –> Apache (8080) –> Redis –> MariaDB

(secure) Nginx (as ssl proxy) (443) –> Varnish (80) –> Apache (8080) –> Redis –> MariaDB

 

Add Link here to Varnish CMS / WordPress, Joomla, Drupal config

Apache / PHP-FPM –> Redis –> MariaDB

 

WordPress Special Notes:

You have to tell WordPress that you are behind SSL and it will function properly. To accomplish this, I use the following code in wp-config.php

if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
    $_SERVER['HTTPS']='on';
}

Be sure to refresh everything once you make you change:

# systemctl restart varnish; systemctl restart nginx; systemctl restart php70-php-fpm; systemctl restart httpd 

You may find yourself needing to download a WP plugin to help with any other issues.

Here are a couple to try:

  • https://mattgadient.com/remove-protocol/
  • https://wordpress.org/plugins/remove-http/

Misc Notes — Found (save) for possible varnish vcl changes




We run Varnish in between an F5 and Apache as well as use Nginx for ssl and load
> balancing in development, in conjunction with WordPress backends. You have to
> tell WordPress that you are behind SSL and it will function properly. To
> accomplish this I’d use the following code in wp-config.php
> 
> if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
>        $_SERVER['HTTPS']='on';
> }
> 
> You can then also set FORCE_SSL_ADMIN and FORCE_SSL_LOGIN however you see fit
> and it should work. I saw some updates not that long ago to support proxy
> headers but don’t believe they are fully supported yet.
> 
> Jason
> 
> 
>> On Nov 2, 2015, at 12:37 PM, Carlos M. Fernández <cfernand at sju.edu> wrote:
>> 
>> Hi, Phil,
>> 
>> We don't use Nginx but do SSL termination at a hardware load balancer,
>> with most of the work to support that setup done in the VCL, and something
>> similar could possibly apply to your scenario.
>> 
>> Our load balancer can use different backend ports depending on which
>> protocol the client requests; e.g., if the client connects to port 80 for
>> HTTP, then the load balancer proxies that to Varnish on port 80, while if
>> the client connects to 443 for HTTPS the load balancer proxies to Varnish
>> on port 8008. The choice of Varnish port numbers doesn't matter, just the
>> fact that Varnish listens on both ports and that the load balancer uses
>> one or the other based on the SSL status with the client (using the
>> command line option "-a :80,8008" in this case).
>> 
>> Then, in vcl_recv, we have the following to inform the backend when an SSL
>> request has arrived:
>> 
>> if ( std.port( server.ip ) == 8008 ) {
>>    set req.http.X-Forwarded-Proto = "https";
>> }
>> 
>> We also have the following in vcl_hash to cache HTTP and HTTPS requests
>> separately and avoid redirection loops:
>> 
>> if ( req.http.X-Forwarded-Proto ) {
>>    hash_data( req.http.X-Forwarded-Proto );
>> }
>> 
>> The backend then can look for that header and respond accordingly. For
>> example, in Apache we set the HTTPS environment variable to "on":
>> 
>> SetEnvIf X_FORWARDED_PROTO https HTTPS=on
>> 
>> I have no knowledge of Nginx, but if it can be configured to use different
>> backend ports then you should be able to use the above.
>> 
>> Best regards,
>> --
>> Carlos.

If you changed your data location for mysql / mariadb, using a new disk and get an error like this after a start or restart:

# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

Here’s the extended status:

# systemctl status mariadb
 mariadb.service - MariaDB 10.1 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2016-03-29 11:49:49 CDT; 3s ago
Process: 5233 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 5219 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=1/FAILURE)
Process: 5185 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 5162 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 5219 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"

Mar 29 11:49:48 db1 systemd[1]: Starting MariaDB 10.1 database server...
Mar 29 11:49:49 db1 mysqld[5219]: 2016-03-29 11:49:49 140524370049152 [Note] /usr/libexec/mysqld (mysqld 10.1.12-MariaDB) starting as process 5219 ...
Mar 29 11:49:49 db1 mysqld[5219]: 2016-03-29 11:49:49 140524370049152 [Warning] Can't create test file /var/lib/mysql/db1.lower-test
Mar 29 11:49:49 db1 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Mar 29 11:49:49 db1 systemd[1]: Failed to start MariaDB 10.1 database server.
Mar 29 11:49:49 db1 systemd[1]: Unit mariadb.service entered failed state.
Mar 29 11:49:49 db1 systemd[1]: mariadb.service failed

It more than likely has everything to do with SELinux running.  To check, do this:

# getenforce
Enforcing

Check to see if this is the issue and temporarily allow us to allow access:

root@db1 /etc/my.cnf.d# semanage permissive -a mysqld_t

Start MariaDB:

root@db1 /etc/my.cnf.d# systemctl start mariadb

Looks good, let’s check the status:

root@db1 /etc/my.cnf.d# systemctl -l status mariadb
 mariadb.service - MariaDB 10.1 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-03-29 11:52:07 CDT; 13s ago
Process: 5233 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 5493 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 5430 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 5407 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 5464 (mysqld)
Status: "Taking your SQL requests now..."
CGroup: /system.slice/mariadb.service
└─5464 /usr/libexec/mysqld --basedir=/usr

Mar 29 11:52:06 db1 systemd[1]: Starting MariaDB 10.1 database server...
Mar 29 11:52:07 db1 mysqld[5464]: 2016-03-29 11:52:07 140638769318016 [Note] /usr/libexec/mysqld (mysqld 10.1.12-MariaDB) starting as process 5464 ...
Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: The datadir located at /var/lib/mysql needs to be upgraded using 'mysql_upgrade' tool. This can be done using the following steps:
Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: 1. Back-up your data before with 'mysql_upgrade'
Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: 2. Start the database daemon using 'service mariadb start'
Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: 3. Run 'mysql_upgrade' with a database user that has sufficient privileges
Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: Read more about 'mysql_upgrade' usage at:
Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: https://mariadb.com/kb/en/mariadb/documentation/sql-commands/table-commands/mysql_upgrade/
Mar 29 11:52:07 db1 systemd[1]: Started MariaDB 10.1 database server.

SIDE NOTE:  Looks like we need to upgrade our disk mounted at /var/lib/mysql now:

# mysql_upgrade -hlocalhost -uroot -pxxxxxxxxxxxxxx
Phase 1/6: Checking and upgrading mysql database
Processing databases
mysql
mysql.columns_priv OK
mysql.db OK
mysql.event OK
mysql.func OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.servers OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
Phase 2/6: Fixing views
Phase 3/6: Running 'mysql_fix_privilege_tables'
Phase 4/6: Fixing table and database names
Phase 5/6: Checking and upgrading tables
Processing databases
information_schema
performance_schema
Phase 6/6: Running 'FLUSH PRIVILEGES'
OK

Now, let’s permanently & persistently allow for SELinux to allow us to write to the new disk:

Start by temporarily putting mysqld_t into permissive mode (should already be there):

# semanage permissive -a mysqld_t

Now, we record what MariaDB is doing and to create a policy to allow that (but nothing else).

Switch SELinux to permissive mode and remove dontaudits from the policy:

# semodule -DB
# semanage permissive -a mysqld_t

Start MariaDB, then use the generated audit log to create a policy:

# grep mysqld /var/log/audit/audit.log | audit2allow -M mariadb_local
# semodule -i mariadb_local.pp

Let’s adjust the file system labelling so that SELinux knows this is MariaDB’s datadir. Check with:

# ls -ldaZ /datadir
drwxr-xr-x. root root unconfined_u:object_r:var_t:s0
# semanage fcontext -a -t mysqld_db_t "/var/lib/mysql(/.*)?"
# restorecon -Rv /var/lib/mysql

You can check what’s in the policy here:

# semanage fcontext -l -C
SELinux fcontext                                   type               Context

/var/lib/mysql(/.*)?                               all files          system_u:object_r:mysqld_db_t:s0 

SELinux Local fcontext Equivalence 

/opt/remi/php70/root = /
/var/opt/remi/php70 = /var
/etc/opt/remi/php70 = /etc

Now, remove the permissive mode for mysqld_t and restore dontaudits:

# semodule -B
# semanage permissive -d mysqld_t