During this amazing journey of developing Affectli, Stefan – Lead Architect and myself are avid supporters of the OSS community, and we thought we would share some insights and lessons learned, as well as our approach to building evolving functionality of the Affectli platform to meet our Customer’s needs – now and in the future.
Glen
Some history
Back in the early 2000s, Mi-C3 had a platform offering that included several modules. Many of these modules were offered by competitors as a single application. A whole company dedicated to delivering and supporting a module similar to what we had on offer. This was a feather in our cap but also a double edged sword – being ambitious, we were at the limit of what technology could deliver i.e. databases, networking, internet speeds, etc.
While we had a single platform with tightly integrated modules that we could offer at a great price, none of these modules excelled compared to other offerings. We mainly covered the basics and offered no bells and whistles. Our resources were spread thin trying to create a feature complete platform and at the same time supporting customers.
We made a couple of decisions and agreed that we wanted to make sure we could offer our clients the best of breed functionality in any module or service, whilst persevering the main ethos of Affectli, of organisations owning their own data, as well as operating on any environment, being device independent and protocol independent.
Coupled with our semantic and ontological approach of building the core engines, we made a strategic decision to use OSS technology that delivers particular value to specific services that are not worth trying to recreate.
Choosing our OSS components and Communities was of utmost importance and strategic interest as we wanted to align and contribute with the best of breed in their field of software, and we have seen huge evolution happen in many areas of OSS that will ultimately benefit our customers.
In the early days, ramping up a development team was tough because the technology stack we built on (InterSystems Cachè) was niche, proprietary and expensive. And while a skilled engineer on this stack could rapidly create a product, they were scarce and expensive.
We knew we had to create something special, starting with a small focused team of 4 to 6 developers. We also knew from previous experience, that a company with about 25 software engineers was not big enough to develop and maintain a complex platform with our intended scope from scratch – it was a tall order.
We started at square 1: Choosing the technology stack.
For me personally, after spending many years honing my skills on InterSystems Cache where we were combining Database, Middleware and Front-End in one place, having to move to something completely new was a difficult choice to make, but I’m glad that I did.
So we had a look at what we wanted to build and what was available within the Developer community.
We needed an enterprise ready database.
DB was our starting point, and we have not regretted choosing PostgreSQL over combinations of MongoDB, MySQL and some other options that were floated at the time.
Over the years since then, new versions of PostgreSQL have added much needed additional features and prevented us from having to deploy other database technologies, except in our Big Data and Analytics offering where compatibility with that ecosystem is better supported by Apache Druid.
We needed a workflow engine.
We previously developed our own custom engine that worked quite well, but then we started looking into industry standards. Out of the many, BPMN2.0 stood out to us, and after some testing, we went with Alfresco Activiti. We decided to host it as a Spring application, running in Apache Tomcat.
We needed a UI Framework and had some previous experience with AngularJS so we settled on the newly released AngularJS 1.5. (Now we use ReactJS)
After many, many lessons learnt, and lots of Dollars spent – not to mention the sleepless nights with this new approach – Affectli was born!
The necessary mistakes
We started development with the most aggressive timeline we could manage. There was no time to think long and hard about things and so we fast tracked the project and it paid off strategically.
But in the end, using open source software became strategic for us. The timesaving methods we used were mostly in our own developed code. The open source software implemented natively in the project, the better our project was for it, because we had less technical debt to refactor later.
Mistake number 1: Roll your own middleware.
We started with a simple Java application that worked by receiving a request from the browser, looking up an SQL statement from a list of files, executing the statement and returning the result to the browser as JSON.
The problem:
This got us to market relatively quickly, but caused us several headaches later on when we had to start adding nice-to-have features to our user interface like advanced filtering, sorting, counting and grouping.
The fix:
We now have an implementation of GraphQL using several open source modules like Apollo client and server.
Mistake number 2: Deploy the demo app to production.
For the workflow engine we needed designer and execution user interfaces respectively.
Activiti came in parts, the Activiti Engine had no user interfaces and the demo application that had the engine embedded in was a “business rules” implementation.
It also came with 2 user interfaces: one process designer and one execution UI for task and process management.
For our needs, we dropped the task and process management user interfaces and we created our own. The rest of it was 80% what we needed anyway, so the fastest way to market was to just use it and modify where required. So we forked it.
The problem:
No upgrade path. After making changes to the code, keeping it up to date with the public repository was a nightmare. We did this once in order to fix a critical bug.
The fix:
Start again and do it right.
We contribute directly to the OSS stack. We do not fork and now follow the community and evolution of the technology.
We built the Affectli engine to handle the changes of OSS technologies, allowing us to replace the OSS technology that does not fit the purpose.
Why open source
A note from Glen –
Mi-C3 is a user of and contributor to open-source software (OSS). Our software and internal tools are built around OSS databases like Postgres, data processing frameworks like Apache Spark, Activiti, NodeJs and frontend libraries like React, just to name a few.
As a commercial software company, we must decide on a case-by-case basis which parts of our software we contribute to the OSS community, and which parts we keep in closed-source repositories.
Depending on the type of software and the nature of the contribution, we have different motivations for pursuing OSS contributions.
What we look out for when choosing an open source project
Fauxpen source
We learned a lesson here after taking the bait, spending time going down the road with a piece of software, until we got to the switch part. There is a trend where companies open source a small core part of a larger software package, (it’s popular to use the GPL license for this), but any meaningful functionality is closed off. Usually, this code comes with little to no documentation and the company would offer that at a premium.
Do your research and take special care to ensure you fully understand their licensing agreements.
One of the projects we were excited about was a chat engine. The server code on Github had a link to the ‘Documentation’ where you can find information about the protocol and encryption, but there is nothing about how to set up and configure a working environment. So instead we went with another engine with ample documentation.
The business model that we appreciate when dealing with companies that are stewards for open source projects are exemplified by 2ndQuadrant who are major contributors to the PostgreSQL project. Here is a small excerpt from their website: “2ndQuadrant encourages and supports the free circulation of knowledge and information. We believe in free access to learning sources, open source software, and alternative intellectual property.”
Permissive vs copyleft licencing
When building for-profit software, copyleft licencing can sometimes be restrictive. When we consider a new piece of open source software, the MIT, Apache and similarly permissive licenses get a quick pass through our filter.
When we deal with LGPL or GPL licenses, we need to ask a few more questions:
“Is this something we will be including in our code or is it a piece of software that we will just be using?”
Another question we ask is: “are we likely to need to make any changes to the software?”
For example: We use Linux extensively but we are not likely to need to make any modifications to it and it’s not something we are linking or embedding in our software, so that gets a pass as well.
I think the copyleft idea is great and provides protection to the open source community for projects like Linux, but the Apache foundation has shown that there is a space where permissively licensed software can flourish and businesses can make use of and contribute to these projects without fear of losing their intellectual property.
Then there are the other licences that raise red flags, various companies have modified their licenses with a few gotchas. We try to steer clear of product specific licenses in general.
Abandoned projects
Github is a great source for finding interesting and useful open source code. A few times we have found some awesome looking projects just to realize a little late that they have been abandoned.
Always look at the commit activity and the issues to see if there is anything happening. Read some of the messages to find out if it is a healthy project and a healthy community. Are there any demos linked in the Readme.md? Are they working? Is there a good amount of documentation?
Hidden costs
While the initial attraction to open source software is often because it is free, as in free beer, you should know that there are hidden costs. In every license agreement there is a clause along the lines of: The software is available “as is”.
Know that your organisation will be paying for it one way or the other. Maybe not in dollars, but you will be paying in time. Time for the engineers and operations team members to figure it out – sometimes by trial and error. Or you may decide to get paid support hours and depending on how critical it is for your customers maybe even an SLA agreement.
So when evaluating a new piece of software, we try to factor in the real cost to the company as part of our decision.
I would be remiss if I did not mention that the Free and Open Source Software (FOSS) movement emphasizes “Free” as in freedom often referred to as Libre.
The GNU free software movement defines free software as follows:
- The freedom to run the program as you wish, for any purpose (freedom 0).
- The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
- The freedom to redistribute copies so you can help others (freedom 2).
- The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.
These freedoms allow the open source concept to function.The licence agreements that do not allow these freedoms should not be called open source licences.
Comparing projects
So you found a few projects on Github that meet your needs. Which one should you choose? Things to consider:
- If applicable, does your ops team need to learn something new to deploy and maintain the solution?
- What languages are used in the project? Do you have inhouse engineers that can understand the code and fix things if needed?
- Is it a popular project? If one of the projects has way more stars, forks and/or contributors than the others, it is more likely to stick around for the long term and have bugs fixed sooner.
- Compare the licenses.
- Who is behind the project? Is there a reputable steward for the project. Look at their website. Are they trying to hide the fact that they have an open source project behind a ‘Pro’ or ‘Enterprise’ version.
- If this is an application, are there readily available docker images? How up to date are the images? Are they maintained in the project itself or by a 3rd party?
- And finally: Try it out! Install the software or do a quick POC. How hard was that to do? Which project was easier to work with? If you got stuck how easy was it to find information on Google?
Conclusion
Since we started working on Affectli, we have added several modules and features and continue to grow the platform.
After the successful launch of Affectli, we now have development offices world wide in 4 locations. With additional presence in South Africa and Nigeria. As we grow, we will build our teams with the open mindset to Open Source as the engineering challenge and problems we are solving in Affectli are substantial.
OSS empowers smaller businesses and levels the playing field to some degree, but can also be extremely valuable for the giants of the industry.
We recently released an open source project: platform-ui is a library of React UI components, based on material-ui. It extends the original with a number of our own components that we find useful. It is released under the Apache 2.0 license and we hope to contribute more to the open source community in the future. Please check it out!