How to develop the right mindset?
Let’s face it, it is 2023, software doesn’t just exist in a vacuum. Software integration (like one of my former colleagues said so accurately) is like backups, there are people who do it and there are people who will start doing it. Sooner or later there is a point in every project where the idea of connecting multiple software systems just magically formalizes.
I’ve been really lucky to work with people in very different roles, different company setups, domains.. etc. and I can still confidently say that there is an underlying trend that just pops up every single time no matter how or when you got involved. So, instead of the ofter really generic approach that guides on this topic follow, here is an attempt of searching for the right mindset when approaching software integration projects through recent real-world examples.
Setting the stage
First of all, this is all going to be really subjective but hopefully (,if nothing else) it will set you on the right pathway to dig deeper in the topic. Integrating software or integrating systems is the act of connecting two (or more) systems so they can exchange information. There are many types of this integration from webhooks, API calls to using middleware.. etc. but our focus won’t be centered around such technicalities now.
Every business maintains some form of list of users, this can be customers, visitors, subscribers..etc. and nowadays most software tools allow for some form of spreadsheet (xlsx, csv) export. So, by downloading your previous contact list from your email software and importing it to the company’s CRM (Customer Relationship Management) system you just created a (painfully manual but) working integration. In reality, you want this data to be kept synchronized and also populated in both directions but I think you already got the concept.
The startup
Many articles on this topic talk about “starting too quickly” or “not planning for scalability” as really costly early mistakes and I certainly agree with them 100% but I would still say that in 2023 if you are a startup software company, you should think about how your solution will fit into your customer’s existing software ecosystem right from the get-go. I would even go as far as saying that this should be one of your key discussions.
We did an extensive research project recently looking into video recording tools for sales, design, and development purposes and it was immediately obvious that customers prefer tools that can work seamlessly with their existing software products right out of the box. Even if you have the best recording quality or the most clever editing tools what you most need is an easy way for your customers to attach the recording to an email campaign or for your developer to add it to the relevant GitHub ticket. If you miss such integrations, you sacrifice usability, and your chances of people actually implementing your tool in their everyday lives exponentially diminish. In many cases, this is so true that even if you have the desktop part figured out but you didn’t think about mobile users people still won’t recommend your solution.
I highly recommend looking into Zapier as it is one of the most popular integration platforms right now. The idea is that most software providers already have a connection developed to Zapier so by building yours, Zapier can act as a middleware where customers can build easy automation so you don’t need to build each of these integrations one by one. It is easy to use and fairly advanced workflows can be achieved.
The enterprise
Unfortunately, I still see an ERP (enterprise resource planning) system implementation fail every single year. Whether it is an in-house built tool or an expensive solution from one of the well-established providers I always feel that the problem comes down to one of two main factors.
First, skipping the POC (proof-of-concept) and with that misalignment with business users comes almost guaranteed. Careful planning, and assembling the right team with the right people are just two ingredients for success but you will also need testing to validate that your idea actually works. Let’s say the management wants to keep track of orders and invoices in a unified way so you want to have financial data in the central system. As a first POC, you could concentrate on automatically invoiced webshop purchases only (and ignore manually invoiced one-off partner orders) and have a batch upload of a selected time period (as opposed to an automated hourly sync) for example. This will give the necessary feedback and confidence to build out the full-fledged solution and again, you can’t save this step.
Secondly, way underestimating the costs and complexity involved. If it is a mission-critical system, you can’t have downtimes, more employees need more training time, transparency means more frequent internal demos along the way and I could go on. It is easy to sort of “overbuy” technology but focusing on the core capabilities that will allow you to get to the project goals the quickest will have much happier results and again, this requires listening to the actual users.
Integration is a form of automation and as such, I often think that it works best if you have a manual process or an established workflow already in place but manual processes are by far the most error-prone in an enterprise environment. Even if automating by building software integration seems like a big upfront cost and human resources seem comparatively more cost-effective, in the long run, you get a return on your investment by minimizing such error factors.
The IT project manager
There is an online clothing store that I particularly like because they have a brilliant free returns policy. Once you decided that the content of the package doesn’t fit, you can just go back to their webshop, select the latest order from your history and click on return. It could not be any easier and I am sure it makes many customers favor them over other stores. This is also a type of B2B (business-to-business) integration where their order management system talks to the shipment management system of the courier company in the background. This not only speeds up the return process, makes it a literal 3-click solution for the customers but also makes the life of both companies easier. However, my address is rather long as the building is separately identified with an extra code and the door also has a longer-than-usual code. This all fits in the 2nd line of the address field nicely in the clothes webshop but as it turns out the courier system can’t accept the long address. As a result, it just rejects the end part which means the courier never finds me when attempting to pick up the return package.
This is to have an example of “bad data”. Bad data can be any inaccurate set of information, like missing data, wrong information, inappropriate data, non-conforming data, duplicate data, and poor entries (typos, variations in spelling, format, etc). Handling such bad data shouldn’t be an afterthought in any data integration project as bad data affects credibility, makes your system unreliable, and can be difficult to repair. So, don’t just focus on the happy path in your data pipelines, validation, and even automatic correction should all be included.
The architect
The architecture and orchestration of such integrated systems is something I would love to dedicate a separate article to as Data Warehouse, Enterprise Service Bus, and Master Data Management… Etc. are all worth a deeper dive but for our current more “holistic” view it is enough to keep two main important things in mind and those are the source of truth and clear responsibilities.
We are working on a really popular Flutter mobile application that has some of its features only available to registered users who are logged in. Pretty standard stuff but the user accounts are actually created on a third-party ticketing website. To complicate things even further, there are account details like “nickname” that are only associated with app users and are not part of the standard ticket account for various reasons. As you can imagine in such situations having a really clear outline of the responsibilities of the connected systems is vital and ideally, you would want to comb out the intertwined blurry lines. When data exists in a similar fashion in various systems across an organization they make for example data-driven decisions challenging as simply not all employees see the same data. Single source of truth (SSOT) architecture or single point of truth (SPOT) architecture is meant to fight against that and create a state where all data can be found via a single reference point.
A possible visualization of this is the Google search engine. Let’s say you want to look for a new local restaurant. In this case, Google tries to be the single source of truth for anything you may want to know about the restaurant. Opening hours, phone numbers, addresses, menu links, ratings, and popular times all appear in one place. In the background, Google brings in data from many sources Maps, the restaurant’s website, Yelp, Google ratings, etc. In this example, your single source of truth for the restaurant data is Google which looks into multiple other systems in the background providing the best customer experience possible.
Finally, let me end this write-up with one of my favorite acronyms, the KISS principle. KISS refers to “Keep it simple, stupid!” reminding us that most systems work their best if they are kept simple and unnecessary complexity should be avoided in 99% of the cases. If this is true for aircrafts, it is for sure applicable when designing your software integration plans.
Contact us and send me your best acronym ideas 🙂
