Running the Sites Backend – Process Progression

Over the years, the how and why of managing the websites’ backend – the files and databases has seen a gradual progression to more complex methods.  I think in many ways it showcases the common route smaller e-commerce / online businesses progress in their processes so I figured I’d write it up.

Single Site – Everything Live

In the beginning, we had single site with everything live on the site.  So any changes we made was automatically on the website as we had to test changes ‘live’.  Bugs, code fixes, new content – it all went up live.  This meant that we had to be careful when we started deploying code and keep all the back-up files on our computer in case something went wrong and we couldn’t figure out the code fix quickly.  On the other hand, it also meant that there was only one site to ever worry about and everyone who worked on it had access to the same files (mostly – see below for potential problems).

This works fine if you don’t mind deploying code and fixes late at night or when you know there are few customers around.  It’s fine if you don’t have a lot of customers or don’t have a lot of big changes to do; but can be a mess if you have a ton of customers at any time of the day or worst; are trying to install a large upgrade / expansion / module.

Oh, the other major issue – all your developers (if you work with more than one) have access to all your ‘real’ databases including customer information. A potential privacy problem.

Staging and Production Sites

A staging site is a ‘fake’ website that (in theory) is exactly the same as your production site.  The staging site can be populated with ‘fake’ database information; reducing privacy problems while allowing you to continue to test code changes.  In addition; because the site is not live your devleopers can put up a partial fix, test it out and then come back to it at a later date (or leave you to do a test).   Timing becomes less of an issue because the staging site can be broken without affecting front-end sales.

Once a change is considered production ready, you can then download the changed files and send the files over to the live site.  This is generally a manual process and one that you have to do yourself.  Part of the reason you implemented this entire process is for privacy reasons.  It makes no sense to ask your developer to do the fix.

Of course, as any IT person will tell you – just because it worked on the staging site doesn’t mean it will work on production.  Sometimes that means a bit of scrambling; but it’s a lot less likely to be a problem.  We’ve been doing this process for the last 3 years or so; bumbling our way through multiple sites, trying to remember to take backups as necessary and keeping multiple versions of the files on our home computer.

Developer, Staging, Production & Version Control

We’ve recently grown-up and moved to a more complex system with 3 sites and version control.

The Developer sites reside on the developer’s server where they test code.  It’s the working version of the site where all the changes are tested in multiple versions till a ‘good’ fix is ready.

Then the ‘good’ fix is sent to Staging, where it’s deployed.  Here, I do the test to ensure there’s no bugs that the developer has missed.  If there are, I send the bug report to the developer who goes back to working on the Developer site before uploading the fix.  If there isn’t, we deploy to Production.

It’s very similar to the above method; except for the addition of Version Control.  We use Springloops for our version control system and can’t be happier.

Version control systems do a few important things for us:

a) it automatically keeps a repository of all files – old and new.  It keeps dates and keeps information about changes so we can ‘roll back’ to an older version with just a click of a button.  No more hunting for files and hoping we had backed it up properly; its all done.

b) deployment of code can be set up to be automatic to Staging servers and Manual with Production servers, while keeping deployment simple.  Quite literally; a click of a button again – so no more worrying if we had missed a file.  The exact same set of files get sent to both; so it removes ‘human error’ from the equation.

c) it allows multiple developers to work on the site at the same time.  Even if a pair of developers download the same file and make different changes, the software will show and indicate any conflicts.  This way, no one developer’s work is ‘over-written’ by accident.

d) it restricts access even further.  With Springloops, we provide access to the repository but not to the actual FTP site. It also lets us invite multiple developers and kick them out easily while keeping track of all the files they’ve touched.

Truthfully, I cannot be happier that we found this solution.  It allows us to get more changes done with new modules and to roll out changes easier.  It’s something I’d recommend to anyone with a website that they have numerous changes on.

Security – Trials & Tribulations

As a business, one of the greatest fears is a security breach that exposes customer financial information.  It’s a nightmare; since being hit by something like this could potentially cripple a business.  We recently had a bit of a scare when 2 customers commented that fraudulent activity had occurred on their credit cards soon after placing an order with us.  Not surprisingly, we decided to conduct a full audit of the site and in the interest of transparency felt we should also write about it here on the blog.

To skip to the end first – there is no security breach on the site.

Background

To understand the story, it’s worth discussing the security procedures that are in-place to keep a customer’s financial information safe.

We do not store credit card information

Those of you who have ever had to edit your order will notice that they generally end up saying ‘Check / Money Order’ on the edited order.  The only time an edited order would say something else would be if the customer had called in to provide us the credit card information again.    This is because we do not store or have access to a credit card once the order is placed.

When an order is placed on the site, the credit card information is sent in an encrypted format to the site and from there, to the credit card gateway who authorises the charge on the card.  We are then provided a token indicating the authorisation for our records.  This allows us to charge a card for the authorised order amount only.  The only credit card information that we store is the card type, the last 4 digits and the expiry date.  None of that is sufficient to run a new charge on the card.

With PayPal of course, all we get is the e-mail address that the payment came from.

Everything is encrypted

The Checkout Page is completely encrypted in a SSL 128-bit encryption (the same method that the big retailers like Amazon use which is basically an industry standard) and anytime we access our backend, all the data passed back and forth is encrypted as well.  So the card is completely secure during transit and on the site.

Regular Scans

Lastly, both our server host as well as our developer regularly run scans to ensure that aren’t any viruses / malware / etc sitting among our files.

The Incidents

Once in a while, a customer contacts us that they have had to change their credit card information due to fraud.  We generally take note of it and run a quick security assessment  but due to the above on-going security procedures it’s generally not likely to have originated from our site.

This time a pair of customers contacted us separately in a very short period, both with very similar stories – initial orders placed very close together, fraudulent activity on the same day, both having orders placed on our site.   That seriously concerned us, enough that we decided to shift gears and focus on a security audit.

The Audit

Since both customer placed the orders remotely, we knew it couldn’t be an HR issue (remember, there’s literally no way for us to get a credit card number unless a customer calls us to place the order over the phone). As such, we knew to focus on our attention on the site and the site code.

We took the audit on in 3 parts.

1) External Audit

We ran the site through a number of external company verifications (e.g. McAffee, Google’s Webmaster, etc) initially to see if the problem was picked up by them. This ensured that no external scripts was being loaded from the site which could have caused problems.

2) Automated File Review

We then began an audit on the files in the site and database. This was an automated process that basically reviewed every file on the site to ensure that it was meant to be there; as well as looking for specific known malicious code.

3) Eyes on Code

Lastly, we put eyes on the code.  Every single file and script that was involved in the process of providing the checkout page on Starlit Citadel was reviewed. Since this is the only location where the credit card information is input, this was the most important ‘fail point’ and thus the extra scrutiny.

In all three tests, we could not locate any potential security problems.While there is never any guarantee, it’s extremely unlikely that we had  a breach in security.  It still is something that had to be done; and I’m open to any other suggestions for things we can do as well to improve security if you have any.  Overall though, it made for a couple of extremely stressful and expensive days.