An end-to-end eCommerce solution requires more than a fancy website

By Tony Marston

9th March 2012

Introduction
The front-end website
The back-end application
The shared database
Back-end processing is not simple
Integrating the front end with the back end
Is there a back-end application that I can integrate with my website?
Conclusion
References

Introduction

The demand for websites on the internet is constantly growing, and so is the army of developers who want to earn money by building these websites. When a company wants an eCommerce website for their business, a website which allows customers to purchase their products over the internet, they turn to this army of website builders and expect an end-to-end solution which satisfies their needs. Yet what they, and even the website builders themselves, fail to realise is that building a sexy-looking website just provides the front end. A proper end-to-end solution must also have a back end, and this back end operates under a different set of rules, is much bigger than you would first think, and requires a different set of skills from the developers.

The front-end website is the equivalent of the front office in an organisation as this is the principle point of contact with customers or clients.

The back-end application is the equivalent of the back office in an organisation. It is less visible to the outside world, but it is still a vital part of the business.

In order to provide a proper end-to-end solution it is a common mistake to assume that you can start with a front-end website and simply "extend" it to include the extra functionality required by the back end. The back end is not an extension to the website, it is a separate application in its own right. It should also be remembered that back office applications existed long before the advent of the internet when all the customer interaction took place in physical stores and not websites. In some cases the back end application is not a single application but a collection of smaller applications each dealing with a separate area, such as "order entry", "inventory" and "shipping". Sometimes this collection of back-end applications has little or no integration, which makes it difficult for users (members of staff) to view data which spans application boundaries.

The front-end website

The main purpose of the front-end website is to look good so as to attract as many visitors as possible. The actual functionality is always quite simple - show the customer what he can buy and encourage him/her to buy it. There will usually be the need to connect to a payment gateway to take the customer's payment, and to communicate with a back-end database, but this communication is usually quite limited - "read products", "add customers", "add orders".

In the very early days all the "purchase" button did on a website was to generate an email. A salesperson from the company would then contact you to arrange payment, either by sending a cheque through the post or by taking your credit card details over the telephone.

What skills does the front-end website require? As it is a web site it requires HTML, plus CSS for styling, Javascript/Ajax for fancy interactions, and lots of sexy graphics plus some Search Engine Optimisation (SEO). There is not much "real" programming involved.

People who call themselves "web developers" may only have a limited set of skills:

  1. Some are nothing more than graphic designers who can produce pretty images. They can show you what they think the website should look like, and describe how it should function, but they cannot actually build anything that works. This means that somebody else has to translate these pretty pictures into working HTML documents and program code.
  2. Some can take a design and build a mock-up of the website in HTML but with dummy data instead of real data from a database. They can also incorporate some CSS and perhaps some Javascript to enhance the look and feel of those web pages. Somebody else then has to write the program code which generates each of these HTML documents, and replace the dummy data with calls to the database.
  3. Some developers can design the web pages, build the HTML, CSS and Javascript, and write the program code which joins those web pages to the database, but they still rely on a "back-end developer" to design the database(s), write the data access code, implement the business rules, and build the back-end application which maintains the contents of the database(s).
  4. Some developers can actually handle both the front end and back end processing, but as each of these ends requires skills which are not relevant in the other, the number of these developers is quite small.

Some non-eCommerce websites make use of pre-written packages which deal with Content Management (CMS) or Customer Relationship Management (CRM) which can be either free or low cost. An amazing number of people seem to think that it is a trivial exercise to install one of these packages and extend it to provide both a front-end and back-end eCommerce solution, but attempting to bend a CMS or CRM package to do what it was not designed to do is guaranteed to end up as expensive, inefficient at best or a total failure at worst.

The back-end application

Anybody who knows business should be able to tell you that although the front office is more visible to the outside world, what happens in the back office is much more complicated, and the front office couldn't function without it. "Taking a customer's order" is just the first step in a series of events which ends with "product delivered to customer". This series of events is commonly known as Order Processing, Order Fulfilment, or even Enterprise Resource Planning (ERP).

Once the customer has entered his order via the front end these order details must be made available to members of staff in the back office so that they can process the order all the way to delivery to the customer. This processing may require several stages, such as "verify payment", "authorise order", "pick from inventory", and ending with "ship to customer". There may be additional stages such as "assemble product" or "manufacture product", or even "obtain product from supplier". It is important that this processing is performed as quickly and efficiently as possible as a dissatisfied customer will be bad for business.

It should be obvious at this stage that the front-end website and back-end application are two different entities.

The fact that the front-end website and back-end application have such significant differences, especially when they are written in different languages, usually means that they have different teams of developers with different skills. Even if the same development language is used for both parts it does not automatically follow that a single developer will have the skills to cover both parts. A front-end specialist will still struggle with the functionality of the back end, just as a back-end specialist will struggle with the fancy graphics of the front end. Developers who can create sexy looking websites are ten a penny, but the number of developers who can design and build entire ERP systems is quite small by comparison.

What skills does the back-end application require? If it is built as a web application then HTML and CSS are required as standard, but the main requirement would be in-depth knowledge of the business processes required in the back-office application, and the ability to design and build the software to carry out those business processes. This comes in several parts:

The shared database

If the purpose of the front-end website is to take orders while the purpose of the back-end application is to process those orders it should be obvious that these two entities must share the same data, which is usually held in a relational database (RDBMS). It would be rather inefficient for the front and back ends to have their own databases as it would require additional processes to transfer the data from one to the other. This would be even more complicated if they had totally different formats and structures. Having said that, it has been known for some companies to use a small MySQL database for the front-end website, then have processes in place to transfer that data to a larger Oracle database for use by their legacy back-end application. While this may be a practical solution for those who have a large financial investment in a legacy back-end application, it is far less attractive to a company which is starting from a clean slate.

You may think that the database only needs to have 3 simple tables - PRODUCTS, CUSTOMERS and ORDERS - but in the real world that idea wouldn't stand up for 5 minutes. In order to cater for the data requirements for a wide variety of organisations which sell different products under different circumstances the experienced developer will tell you that each of those areas has such a wide ranging number of possibilities that they require separate databases, not just separate tables. Let me explain what I mean:

The PRODUCT database may require the following tables:

The CUSTOMER database may require the following tables:

This can become complicated if a customer can be either a person (as in a business-to-consumer (B2C) website) or an organisation (as in a business-to-business (B2B) website). If the customer is an organisation then the system will need to know the name of the contact person at that organisation.

With a B2C website it is usual to take the customer's payment before the order is processed, but with a B2B website it is usual to take the order without payment, then send the customer an invoice with payment due within a certain time limit.

What happens if you also have to deal with suppliers? Do you have separate tables for customers and suppliers, or can you merge their details into a single table? What happens if some people/organisations can be both customers AND suppliers?

The ORDER database may require the following tables:

As you should be able to see, the actual data requirements cannot be met satisfactorily with just a small number of simple data structures. You have to have a database design that allows as many options as possible, and software which can deal with all of those options. You may not think that all of those options are actually necessary, but those of us who have been working in the real world for long enough know that an organisation's requirements have a habit of changing and becoming more complicated over a period of time, so it can be worthwhile to have a system which has a lot of the common options available from the start so they can be quickly turned on as and when required.

As the shared database must cater for the needs of the back-end application, which is much more complicated than the front-end website, it follows that the database should be designed to satisfy the needs of the back-end application, and the front-end website should be altered to deal with that design.

The back-end application may also need additional databases for additional functionality, such as "invoicing", "inventory" and "shipments".

Back-end processing is not simple

Building a website that looks sexy is one thing as it is all glitz with very little functionality that requires "real" programming. This is something that a lot of website designers can do with their imaging software and a bit of fancy CSS and Ajax (Javascript). Building a back-end application is totally different as the glitz is irrelevant and the functionality is everything. You have to be able to design and build a database to hold all the possible variations in data, then you have to design and write the code which manipulates that data in the way which is most effective for the business. You have to understand the business processes which are involved, then be able to translate all those business processing rules into effective code.

Creating a website which takes customer orders is quite simple, but can you write an application which progresses those orders all the way from order authorisation, picking from inventory and then shipping to the customer? If you have customer shipments then you must be able to deal with customer returns. If you have inventory then how is that inventory replenished? Do you raise purchase orders on suppliers, do you manufacture parts or assemble them? If you deal with purchase orders then you must also have a mechanism for dealing with purchase returns.

If you have inventory then you need to record what stock is stored in which container at which location, which may be a different room, a different building or even a different address. You will need to record all receipts into stock as well as issuances from stock, and allow for variances following a stock check. You may also need a means to reserve stock so that when an item is selected for one sales order it cannot be selected for another. You may also need mechanisms to move stock from one container to another within the same location, or between different locations by means of transfer orders (both in and out). You may also need a mechanism to warn you when the on-hand quantity of certain items goes below a certain level so that you can start the replenishment process.

Once you have taken a sales order there may also be a requirement to turn this into an invoice, convert this invoice into a PDF document and email to the customer. If you have invoices then you will also have to deal with credit notes.

Do you want the system to track all activities for each customer? Do you want the system to send out emails? Do you want the system to act as an email client and track all communication with each customer? If you do then you are venturing into the realm of Customer Relationship Management (CRM).

Integrating the front end with the back end

The front-end website and back-end application are separate entities, with the only common ground being the shared database. This is shown in Figure 1:

Figure 1 - Separation of front-end website and back-end application

front-end-back-end-01 (2K)

In this example the front end and back end are entirely different applications, almost certainly developed by different teams of developers, with different skills, and more than likely written in different programming languages. If there is any business logic which is required in both parts, then the programming code to implement that logic will have to be duplicated as it cannot be shared.

This inability to share code between the front end and back end results in an expensive duplication of effort. It may also lead to errors if any changes to the business rules are not synchronised properly in the two applications. This situation can be alleviated somewhat if the two parts are written in the same language, and if the back-end application has been developed around the 3-Tier Architecture (see Figure 2)

Figure 2 - The 3-Tier Architecture

infrastructure-03 (9K)

In some large organisations it has been known for them to have separate teams of developers to work on each of the different layers even though they are using the same programming language. This is because the presentation layer is concerned with HTML, CSS and Javascript, the data access layer is concerned with SQL (and possibly database triggers and stored procedures) while the business layer is concerned with enforcing the business rules, data validation, flow control, etc, and does not deal with any HTML, CSS, Javascript or SQL.

With this architecture any one of the layers can (in theory) be modified, or even replaced, without having to make corresponding changes to any of the other layers. This means that the front-end website can have its own presentation layer (User Interface or UI) and share the same business layer and data access layer components that the back-end application uses. The business layer does not know (or even care) which presentation layer is communicating with it - it simply receives requests and returns the responses, and it is up to the presentation layer to deal with each response as it sees fit. This is shown in Figure 3:

Figure 3 - Different Presentation layers sharing the same Business & Data Access layers

front-end-back-end-02 (5K)

In this scenario the code within the front-end website is concentrated around the fancy user interface, and when it wants to communicate with the database it accesses one of the many business objects in the back-end application with a simple method call, such as $object->getData() or $object->insertRecord(). This means that the front end is as light as possible, with all the heavy lifting, data validation, business rule processing and SQL query construction handled entirely by the back end business layer components. As all the business rules are handled in a single place, the business layer components, then changes to business logic are made in a single place and are automatically made available to both presentation layers.

For this to work all the application code must exist on the same server.

If the back-end application has been enabled for web services (ie. it has a component which accepts web service requests) then would be possible to change the front-end website to include a web service client in order to communicate with the back-end web service server. This is shown in Figure 4:

Figure 4 - Linking the front and back ends with Web Services

front-end-back-end-03 (12K)

This method has two distinct advantages:

Is there a back-end application that I can integrate with my website?

All this talk about integrating a front-end website with a back-end application is all well and good, but does such an application exist? Well, now you come to mention it, yes it does. There is a product called TRANSIX which has the following characteristics:

The TRANSIX application was also built around the 3-Tier Architecture, so it offers the benefits shown in Figure 3.

The TRANSIX application also includes a collection of web services, both client and server, so it offers the benefits shown in Figure 4.

If TRANSIX has been built as a package, what are the possibilities for customisation when a client has non-standard processing requirements? The core code is sacrosanct and remains the same for everybody, but custom variations can be introduced by means of plug-ins. These are scripts which can be dropped into any of the separate plug-in directories, with separate directories for each client. With these plug-ins it is possible to change a screen layout, a PDF report layout, or even the code which is executed for certain events. Changes to custom code may be implemented either in conjunction with or instead of the corresponding core code.

Conclusion

I hope this article has shown you that building an end-to-end eCommerce solution does not stop with a fancy looking website. The front-end website is just the tip of the iceberg with all the heavy lifting done by the back-end application. Think of the front-end website as the showroom in a high street store. It is designed to attract customers and make sales, but regardless of the activity in the showroom you should realise that all the important work takes place behind that little door at the back labelled "staff only". It is the work done by the support staff in the back office that makes the activity in the showroom look effortless, and without it the glitz of the showroom counts for nothing.

The front end website and the back end application are totally different entities with different goals. In some organisations they are built in different languages, which automatically implies that they will have two separate development teams. Even if the two parts are written using the same language it would be naive to automatically assume that they can be built by the same developer(s). They are still different types of application with different goals. Building a sexy front end and a functional back end require totally different sets of skills, and it is extremely rare to find a single individual who has the necessary skills to cover both areas.

The most workable solution for organisations who are venturing into eCommerce and who do not have a legacy back-end application to support would be to find a back-end application that can easily be integrated with a front-end website. When the website is constructed, usually with dummy data to start with, it can easily be modified to communicate with the TRANSIX database. This means that as soon as an order is created in the website it instantly appears in the back-office application and can be processed to completion. This procedure has been employed for organisations in fields as far apart as jewellery, baby clothes and beauty products, so it is a proven product with a proven track record and not just vapourware.

But what if an organisation has an existing website and small but inefficient back-end application? This is slightly more involved, but still possible. It just requires two steps instead of one:

  1. Transfer existing data from the legacy database to the TRANSIX database(s).
  2. Convert the website functions to access the TRANSIX database instead of the legacy database.

This later procedure, the replacement of a legacy back-end application, has actually been done with TRANSIX, so it is not unknown territory.

If you would like more details of the TRANSIX package you can contact me using this link or this link.

References


© Tony Marston
9th March 2012

http://www.tonymarston.net
http://www.radicore.org

counter