At Ethos we are building a distributed mortgage origination system and in mortgage there is a lot of
different user types with processes that vary depending on geography. One of our ongoing discussions is about how much of the logic resides in code vs. being in a workflow system or configuration. After researching this topic for a bit, I have arrived at a conclusion that the logic should live outside of code very infrequently, which might come as a surprise to a lot of enterprise software engineers.
Costs of configuration files and workflow enginesFirst thing that I assume is true is that having any logic outside of the code has costs associated with it. Debugging highly configurable system involves not only getting the appropriate branch from source control, you also need to make sure that the right configuration values or the database. In most cases this is harder for programmers to deal with. In many FinTech companies where the production data is not made readily accessible to developers this provides additional barrier to solving customer issues.
Writing unit tests also becomes harder, to spin up a unit tests the data living outside the code needs to be populated. With an elaborate configuration you also need to test the different permutations that the data can take on, which can very extensive.
IT AdministrationThere are systems that need to support a variety of different workflows depending on the organization or even a particular transaction. A good example of such a system is a CRM. CRM by itself is generally a customer database with a nice interface, but sales and support patterns vary greatly across organizations. The type of data that is being tracked is also very different among the different customers. A big audience in a system like that is an IT administrator. CRM, ERP, LMS and other categories assume that upon purchasing a “platform” you will have someone go in and configure it to fit a particular business need.
Systems that have this type of flexibility requirements cannot have all the logical forks in code, there is a need to create an interface for administrators to configure the system. If you have this flexibility and a nice, well documented user interface to work with it then the system gains flexibility necessary to support a variety of different scenarios.
The key in these scenarios is to know the true cost of providing every configuration switch or workflow support: documentation, debugging, export/import, permissions and versioning of such a configuration are probably going to be needed. A lot of people come short of this, they create a configuration file or a workflow database table, but no way to work with it or support it. As a result, you pay all the costs but don’t get the benefits: the only people who can configure such a system are the developers who wrote the code that processes the configuration.
SaaS vs On-PremSome systems have a requirement to be on premise. When an upgrade to the logic requires someone to take the system down, upgrade the logic and possibly migrate the data, it might make sense to take some of the logic out of the code into a configuration file. Managing different versions of the solution that might have different business processes could be very hard, you might need to provide the configuration, import/export, admin interfaces and debugging to such a system anyway.
Systems that are deployed and managed by the vendor “in the cloud” can get away without providing an administrative interface. The engineers can update the code when the business requirements change, given that it is not scenario specific. There is also no need to support multiple versions of the code when you deploy it to your own infrastructure.
How frequently does the workflow change?One of the key data points on configuration vs code is the frequency when the logic changes. Some logic just doesn’t change that frequently. Here is an example: password reset and new account setup are scenarios that are well understood and rarely change. The content and design of the e-mails might be different and might also be stylized based on what tenant is being used. This is a case where most of the engineers split the configuration vs code in the right place: code for handling the reset/new account logic and configuration for the e-mail template. The idea is that the password reset logic is not changing that frequently, but the e-mail template might.
What technology is used?Last piece of the decision is the technology that is being used. Systems that are based on interpret languages often ship as source code and there is no reason to create a separate set of files to configure it. A famous example of such a system in settings.py for Django. The variables for settings can be right inside the Python code file without a need to create a config file. Settings file can be extended to take environment variables for some settings that are expected to change such as debug level, but not for everything.
Interestingly enough some of the compiled languages like C# have moved to a deployment model where you do not need to compile them, you can just issue ‘dotnet run’ and the program is going to compile and run without having an explicit compilation step.
If you are using a technology stack that doesn’t require you to compile into binaries (or if that happens behind the scenes), it might be just as easy to keep the configuration values, and settings in a code file in a known location.
SummaryGiven the tax of configuration it’s very important to ensure that you weigh pros and cons of supporting logic external to code in your system. I’ve seen people too many times err on the side of providing an extensible system and ending up in a system that was slow to develop, impossible to debug and hard to manage. The configuration points provided were never used by IT Administrators because the administrative interfaces were poorly documented and the engineers constantly had to get into customer data to make sense out of what the system was using.
Configuration and workflow are powerful tools, but just like any powerful tool it’s easy to shoot yourself in the foot.