Page tree
Skip to end of metadata
Go to start of metadata

Third Light Synchronisation Server requires a licence key and support and maintenance. It is required for every Data Center Edition deployment. Please [email protected] if you need a license.

Introduction

Your Third Light Chorus server is designed to operate in a pair, with a master and a replica. To make proper use of this feature, two separate servers are required. The servers are usually in distinct locations. Replications between the servers occurs over an SSH (encrypted, compressed) network connection, and is suitable for use over public internet connections, for example.

 

  • The Primary Server is your main, production hardware running your Third Light Chorus system.
  • The Synchronisation Server is the additional backup or mirror server which replicates from the Primary Server.

In the event of a disaster, a web interface on the Synchronisation Server provides a single push-button process to activate the server (make it live). At this stage, all synchronisation is stopped and the two servers are disconnected from each other, to prevent any problems. 

The Synchronisation Server requires its own license key.

A web-based administration tool is provided on the Synchronisation Server, allowing you to configure the master-slave cluster, check that synchronisation is active, and, if required, to make the Synchronisation Server into a production system. 

The Synchronisation Server is generally of a slightly greater capacity than the Primary Server, but lower CPU/memory specification as it is not used by users in normal operation.

The installation and configuration required is discussed, next.

After a disaster has occurred and the Synchronisation Server is live, you may wish to later switch back (for example, after repairs or replacement have taken place).

In this case, the recovery procedure is to install the Synchronisation Server packages on the old Primary Server (in effect, reversing the roles of the two servers). Then, following the same process of using the push-button process to activate the server, the roles are reversed. For maximum safety and reassurance we recommend involving Third Light Support where time permits.

Configuring the Synchronisation Server

For simplicity, we will assume the host name of the Primary Server is chorus-primary and the host name of the Synchronisation Server is chorus-secondary. You should substitute the hostnames or IP addresses of  servers where chorus-primary and chorus-secondary are mentioned.

Before you start, ensure that Chorus has been installed on the Primary Server (by following the installation guide PDF supplied with your purchase). You will need the following details from that installation:

  • The root password for the Primary Server
  • The hostname or IP address of the Primary Server

The Synchronisation Server must be able to reach chorus-primary on TCP port 22, SSH.

Next, install the Chorus software onto the Synchronisation Server in the same way that you installed the Primary Server. 

  • The initial installation process is identical to installing a Primary Server, and is described in the Third Light Installation Guide (PDF). This is available by emailing [email protected] if you have misplaced your copy. 
  • The Synchronisation Server can be added at any time provided the details for the Primary Server, above, are known. 
  • No interaction with the Primary Server is required as all configuration is managed via the Synchronisation Server's own user interface.
  • When you are installing the Synchronisation Server, you will need a unique license username and password. Do not enter the same license details as the Primary Server. 
  • The Synchronisation Server will  have its own hostname, IP address and passwords. You will need to keep a record of these. 
  • All other settings requested during the installation of Chorus are the same as you entered for the Primary Server.
  • When you have completed the standard installation of the Third Light software to your Synchronisation Server, you will be able to login as the root user. You can do this interactively via a local console, or by using SSH and using the username "root". 

Important: check that you are logged into the Synchronisation Server (chorus-secondary), not the Primary Server (chorus-primary). You do not need to login to the Primary Server to complete this installation, and should never install chorus-sync on the Primary Server. It is only needed on the Synchronisation Server.

After logging into the Synchronisation Server, run these commands:

apt-get update
apt-get install ims-sync

Follow the prompts to install the packages for ims7-sync. A number of dependencies will also be installed.

Exim provides local SMTP outbound mail handling for your server. Next, configure the mail transport agent (Exim) so that any email alerts from the server can be delivered:

dpkg-reconfigure exim4-config

Once the package has been installed and Exim is conifgured, reboot and then you can configure the Synchronisation Server via its user interface. You can do this by logging into the built-in web server.

Logging into the Synchronisation Server

Using your web browser, navigate to https://chorus-secondary:8443. 

 Please note that you will need to specify HTTPS in the address bar, as this is a non-standard port, and you may need to accept HTTPS (SSL) certificate warnings in order to open this page as a self-signed certificate is used. If you have not installed the ims-sync package (see above) then no response will be received.

If you have previously configured the Synchronisation Server, you will see a login screen. Enter your Synchronisation Server license username, and then enter the password. If you do not have these details, please contact [email protected] 

If you have not previously configured the Synchronisation Server, you will reach the initial configuration form, as follows:


Enter the following information to complete the form:

  • Primary Server hostname or IP address: enter the address of your Primary Server. This must be reachable via TCP/IP from the Synchronisation Server (ports 22, 80 and 443 are used).
  • Primary Server root password: the root password you set in the installation process.
  • Synchronisation Server license name: the username of your Third Light Synchronisation license key. Take care - please do not enter the license of the Primary Server. If you do not have this information, please contact [email protected]
  • Synchronisation Server license password: the password for your Synchronisation license key.

The optional settings are normally correct and should not need changing unless you have specifically changed the SSH port number or require a longer timeout. Otherwise, enter port 22 and a timeout of 2 seconds, then click "Submit".

The Primary server database password is blank by default. Only enter a password here if you know that one is required to authenticate with the MySQL root user account.

If all of the settings are correct, you will reach the main Synchronisation Server user interface, otherwise an error message will be displayed to help you.

Using the Synchronisation Server web interface

Configuration tab

The Configuration tab contains all of the settings for your Synchronisation Server, and must be correctly setup before your first backup will successfully transfer from the Primary Server.

The only settings that must be set are the Global Settings. You do not normally need to edit any Advanced Settings.

Configuration: Global Settings

Setting labelDescription
EnabledRuns daily via Cron, or when manually initiated. Disabled when the Synchronisation Server is in production use.
Maximum Generations

Keep roll-back data for this number of days. Typically allow 0.5% storage overhead per day, although in delete-heavy environments this can be much larger

Primary Server IP address or hostnameThe hostname (or IP address) of the Primary Server to back up.
Database: MySQL root PasswordIf you have configured a root account password for MySQL on the primary server, it is needed here to import the database. This value is normally blank.
Notify: Error E-Mail AddressIf an error occurs during backup, send a notification to this address

Configuration: Advanced Settings

Do not change Advanced Settings unless you are certain that the setting is required. No changes are required on this panel to make the Synchronisation Server work as designed.

Setting labelDescription
Directory Settings
 Additional Datastores

By default, just /space will be backed up. If you have additional datastores (/space2, /space3,...) on the primary server, enter them here, in the format "space2 space3"

 Roll-Back Path

The directory to store roll-back data for the main content store. You may wish to change this if you have a dedicated mount point for roll-back data

 Database Archive Path

The directory to store old snapshots of the database. You may wish to change this if you have a dedicated mount point for roll-back data

 Database Snapshot Path

The directory on the Primary Server where database snapshots are made by hotcopy. It is normally only necessary to change this if the company name was configured differently on the Primary Server and the Synchronisation Server during installation

 Date Format

Rollback data and databases are stored in directories named according to this, in GNU date format (see https://www.gnu.org/software/coreutils/manual/html_node/Examples-of-date.html)

 Additional Directories

If you need to back up directories other than ims, archives or uploads from within each /space directory, specify them here. For example, to back up /space/flatcopy specify "flatcopy"

Sync event reporting key pathThe Synchronisation Server registers a key with the Primary Server to allow it to report backup progress. It is stored in the specified file.
Global Settings
PID FileThe backup process stores its process ID in a file to allow monitoring and to protect against concurrent runs
Assume storage appliance replicationIf you have a replicating pair of filers, such that the /space<n> mounts on this server have content that implicitly mirrors that on the Primary Server, enable this option. In this event, the Synchronisation Server only considers database and application state, and will not perform any bulk data transfer. Do not enable this option if you do not have a mirroring solution.
Network
SSH PortIf you have SSH running, or forwarded from, a non-default port, specify it here
Disable Ping DiagnosticThe Synchronisation Server pings the Primary Server as an initial test of its availability. If your network environment blocks ICMP ping then you will need to select this option to bypass that check.
Bandwidth Limit

During network copy operations, limit data transfer to the specified rate, in kBps (not kbps). Leave empty to use max available bandwidth

Use Transfer Compression

If enabled, network copies use GZIP compression. Typically this is not a useful optimisation, since most modern file formats include internal compression, so the achievable reduction in bandwidth is small compared with the CPU overhead

Copy Retries

The number of times to retry a copy if the network connection is dropped

rsync Arguments

For advanced users, you can modify the rsync flags used by the data transfer (see https://download.samba.org/pub/rsync/rsync.html)

Log tab

The Log tab contains a condensed or abridged record of actions taken (excluding low-level debugging information). For example, it will show when an action starts and when it succeeds, as well as logging any failures. The contents of this log are emailed to the "Notify" address on the Configuration tab in the event of any errors. 

For low-level logs, go to the Actions tab and look under "Actions Log". This information can be cut-and-pasted to Third Light in a support ticket and will expedite your ticket being resolved in most cases.

Actions tab

Actions: Triggering a manual backup to the Synchronisation Server

Click the button labelled "Start a new backup", and provided a backup is not already in progress, the Synchronisation Server will immediately begin the process of connecting to the Primary Server to create a new backup. When the backup is complete, a green banner will be displayed at the top of your browser view to indicate that the backup succeeded. If you see any error messages instead, go to the "Log" tab or the "Actions Log" on the Actions tab to see more details.

Actions: Failing over over to the Synchronisation Server

Clicking the button labelled "Emergency Activation: Make the Synchronisation Server live now" will stop the Synchronisation Server, and convert it to a production Third Light Chorus server based on the last backup and database replica copied from the Primary Server. You can login to the server by accessing it as https://chorus-secondary (HTTPS is used).

You can also update your DNS servers to point the production hostname of the Primary Server to the Synchronisation Server, which is normally needed to ensure that users reach the correct IP address and do not see HTTPS certificate warnings.

Actions: The Actions Log

When the Synchronisation Server is busy completing a backup or becoming active, you can see what the low-level operations are by looking at the Actions Log on the Actions tab. In the event of any problems, error messages or other complications arising you can supply the contents of this log to Third Light support at [email protected], in order to facilitate a swift resolution of any issues.


Disk Status tab

The Disk Status tab shows a report of the storge available on both the Primary Server and the Synchronisation Server. It is essential that the Synchronisation Server has at least the same amount of disk space as the Primary Server, and ideally, a further margin of storage to allow for daily roll-backs. 0.5% of storage per day is recommended as a typical margin. For example, if a 28-day roll-back window is desirable and the Primary Server has 1000GB of storage, then you should ensure that the Synchronisation Server has approximately 1140GB of storage.

Note: the Disk Status page will not be available immediately after booting your new Synchronisation Server. The information required is gathered shortly after the Synchronisation Server starts up.

Checking the status of the Synchronisation Server

From the Primary Server

The primary server features a status page so that the success of failure of backups can easily be seen:

Configuration > System Administration > Backups

In order that the Synchronisation Server can send a status report to the Primary Server, it must be able to contact chorus-primary on port 443 (HTTPS). An error regarding an 'unexpected status' may be seen if neither of these ports are open.

From the Synchronisation Server

You can  see the status of the system from the Synchronisation Server by using the Log tab. Any errors or warnings are also displayed in a banner across the top of the page.

Background Information for Systems Administrators


The following information is supplementary and should not be required except in very specific circumstances. Where possible, use the Synchronisation Server user interface to manage your backups and contact [email protected] before making low-level adjustments.

Configuration Files

The configuration of your Synchronisation Server is in the following file:

/opt/thirdlight/etc/ims-sync.ini

This file is read from the Synchronisation Server user interface, and changes committed from the user interface are written to the configuration file.

The configuration fields are as follows:

IMS_MASTER

The IP address or hostname of chorus-primary. chorus-secondary must be able to contact chorus-primary on ports 22 (ssh) and 443 (HTTPS).

MAX_GENERATIONS

The number of days of backups to keep. Additional backup generations require more storage.

DBROOT

Default: /space/db_backups

This directory contains the database backup snapshots taken from chorus-primary.

Ensure that this location can store the size of your database (/var/lib/mysql on chorus-primary) multiplied by MAX_GENERATIONS.

ROLLBACKROOT

Default: /space/rollback

This is the local directory to use to store data rollbacks. These can be used to incrementally restore the data directories to any of the last MAX_GENERATIONS days. This should be on a large volume, as
it will contain all data deleted in the last MAX_GENERATIONS days.

DIR_DATE_FORMAT

Default: %F_%H-%M

This is a date format string for GNU date (see date --help). It is used to generate daily directories for backup generations in ROLLBACKROOT and DBROOT. In normal use you should not change this value.

EXTRA_TOPDIRS

By default, the synchronisation process copies all Third Light directories. You can add additional folders here (relative to /space).

EXTRA_SPACEDIRS

Optional parameter for additional space[n] directories. Enter as "space2 space3" etc. /space is always included by default.

Note that the size and layout of /space directories on the Synchronisation Server must match, or exceed, the Primary Server.

MYSQL_PASSWORD

The MySQL root password for chorus-primary

HOTCOPY_DATA_DIR

This is the directory on chorus-primary to which MySQL database snapshots are saved. It defaults to /space/ims/<sitename>/hotcopy, and should not normally need to be changed.

ERROR_NOTIFY

Default: [email protected]

This is the e-mail address that is notified if the backup encounters an error condition (default: [email protected]). You must ensure that email delivery is configured and working (e.g. dpkg-reconfigure exim4-config).

READ_ONLY_SLAVE

Default: (not set)

If the Synchronisation Server uses a read-only mount, replicated by an external process (e.g. NetApp replication), the synchronisations process should not copy data from the master. In this case, set the READ_ONLY_SLAVE option to 1 to perform additional actions on the master, and to prevent attempting to copy data under /space.

DO NOT enable this option in the absence of external replication, as it will disable data copying.

COMPRESS

Default: (not set)

To enable compression on data transfers, set this to a value of 1. This enables gzip compression. This is not usually beneficial as the data being replicated is generally already in a compressed format (e.g. GZip or JPEG files).

NO_PING

Default: (not set)

Set this option if running the sync manually errors with 'Master server not reachable' due to inability to ping chorus-primary from chorus-secondary.

Running a manual sync from the command line

Occasionally it may be useful to run the sync manually from the command line:

/opt/thirdlight/sbin/sync_server

Roll-Back Files

Roll-back files are stored on the Synchronisation Server. The Roll-back files are located in /space/rollback, and contain a record of changes made each day. These files accumulate with each backup. The roll-back files can be re-integrated with the most recent content to transform the storage on chorus-primary back to an older state. In effect, the roll-back files can be used to undo whatever changes have been made. This is a powerful and useful feature which is achieved with minimal storage overheads, as the roll-back files only record differences.

You will need to have access to the roll-back files for the period you want to undo. To check which roll-back files are available, go to /space/rollback and use ls -1tr to list the contents in reverse order. The last directory will be the most recent roll-back. The date and time is encoded in the folder name.

2017-05-09_02-45 3 days ago
2017-05-10_02-45 2 days ago
2017-05-11_02-45 1 day ago

You can restore the any number of roll-backs provided they are applied in sequence (e.g. to roll back three days, execute the rollback scripts from today, then yesterday, then two days ago). You can only apply roll-back files in strict reverse order. To restore 1 day, in this example we would use the 2017-05-11_02-45 roll-back files. Inside this directory are ready-made Shell scripts (.sh extension) which automate the restoration of that roll-back, to save you time.

On chorus-primary, as root, run the commands that end in '.sh'. Please note that you may not necessarily see space2_ims.sh, space3_ims.sh depending upon how may data stores are configured.

/space/rollback/2017-05-11_02-45/space_ims.sh
/space/rollback/2017-05-11_02-45/space_archives.sh
/space/rollback/2017-05-11_02-45/space2_ims.sh

Each script issues the commands necessary to restore the roll-back. Depending on the size of the roll-back and the size of your Third Light site, the ims.sh operation may take a while to complete while the other operations will be fast.

Once the process of re-integrating the roll-backs has completed, you must restore the database backup for the same point in time.

We provide a script called restore.sh which will inspect and recover your Third Light site, provided that the /space backup is a complete and consistent backup.

To find restore.sh, go to /space/ims/sitename. If you are unsure what sitename has been used, run this command:

ls /space/ims/

You will see a themefiles directory and at least one sitename.

Next run /space/ims/sitename/last-good/restore.sh.

You will be asked to confirm that you wish to proceed, and the date of the backup will be displayed:

This process will restore the database and system state from the backup taken on 2012-03-06_00-24
This restore does not include file content - please ensure that all datastores are mounted or restored
Any existing database or configuration will be replaced with the backup.
Do you wish to continue? (y/N)

Depending on the state of your server, you may see packages downloaded, being configured or replaced. Once this process is complete you will see this message:

Your Chorus server has been restored successfully.

The Third Light site is now ready to use. If you do not see this message, there may be more complex underlying dependencies or changes required. It is usually best to refer such problems back to Third Light. Please ask for help by emailing [email protected]

Syslog Integration

The Synchronisation Server logs progress and events to syslog as ims-sync, using the local6 facility. If you have a central syslog server, you may wish to configure the system to send these messages there.

Third Light servers use rsyslog, which is simple to configure for remote logging. For example, the following will forward all log events to syslog.mydomain.com UDP port 514:

cat > /etc/rsyslog.d/ims-sync.conf <<'EOF'
$ModLoad imuxsock

$ModLoad imklog

# UDP forwarding mode
local6.* @syslog.mydomain.com:514

EOF

/etc/init.d/rsyslog reload

Severity levels of error, warning, notice and info are used.

  • error: The sync/restore process failed
  • warning: Events such as sync being disabled, another process already running, or a diagnostic disabled by configuration
  • notice: Successful completion
  • info: Additional information about progress or active tasks
  • No labels