Perl in a Business Application - Musings of an Architect?

Everybody knows that Perl is not the right language for a large scale enterprise application. This is common knowledge, right? But why is that? Explanations are as many as there are people explaining. Everything from "it's a compiled language, therefore slow" to "its free syntax breeds discoherence" to "Perl developers are horrible individualists".

Well, I didn't believe this, and I went on to help in a startup which wants to build some fintech systems, the first aim of which is to integrate with Finnish banks and collect daily payments from a customer's bank account.

It was decided to use Perl as the core language. If Perl is (was) good enough for Goldman Sachs and Morgan Stanley it surely is good enough for us. So off to build a framework!

Two Failed Attempts for System Architecture

We decided to do the web part with Dancer2 and build our own object system with mappers to read from and write to database and a clever filing metaphor with a class called BusinessObjectManager which creates, stores, restores and retires (removes) one object at a time. I had previously worked with a similar kind of metaphor in a C++ system. That was, of course, much more rigid, whereas with Perl we didn't seem to be able to create the safeguards we wanted to prevent future developers from abusing the system. The design just grew more complicated. I considered employing Moose but then instead took a whole different approach.

An approach familiar from Java web programming, Java internal services, which concentrates on business processes instead of business objects. I built an example service called SimulateBank with a simple structure: At the top was Dancer2 SimulateBank::Web package creating the WWW interface, this used SimulateBank::Client package which in turn called the JSON REST Api. Api was done with Dancer2 using SimulateBank::Api package which in turn used internally SimulateBank::Service which spoke to database directly. SimulateBank::Service implemented "services" like find_deposits, create_deposit, report_deposit and cancel_deposit. These are processes which codify the business rules of the company. We do not care about objects, we only care about interfaces and processes.

This second approach was much more convenient and efficient. However, it still required code duplication and syncronization between Client and Service. What's more, I wasn't happy with it because it didn't feel "Perlish". I was resonably happy with the "super-structure" of the code, the division between Web and Api. But I felt I wasn't using the possibilities of Dancer2 framework with its plugins, like Dancer2::Plugin::Database and Dancer2::Plugin::Queue. I felt I was doing the same job twice when implementing my own interfaces to database and message queue instead of using the readily available Dancer2 facilities.

├── SimulateBank
│   ├── Api
│   │   └── Transactions.pm
│   ├── Client
│   │   └── Transactions.pm
│   ├── Common.pm
│   ├── Service
│   │   └── Transactions.pm
│   └── Web
│       └── Transactions.pm
└── SimulateBank.pm

And then it struct me! I had looked at the whole problem from the wrong angle. My approach was code first. I tried to create a perfect structure for the future Perl programmers to use.

Towards a Perlish Approach

Perlish approach is result oriented. After all, do we not pride ourselves on using a language which is fast to program with? Software only matters if it's put into production. But what about all that horrible Perlish hacking, the quick-and-dirty way?

That is the Perl way! To create a working solution today. Not to worry about tomorrow.

In my experience the most far reaching problems in software development are not done by programmers, or they are done by programmers when they have been forced into roles that should be done by others, such as database designer, integration architect and user interface designer. These are the people who should do the worrying. They are paid to design long-term solutions!

Business App Blues

The purpose of most enterprise applications is to collect data, and then deliver or distribute it, or act upon it. So is ours. From the very beginning we started to plan our datamodels and the resulting database schema meticulously. For instance, the application has only SELECT and INSERT access to many of our tables to prevent the loss of past state information. The whole schema has only one sequence and its value is inserted in every table so that the whole flow and order of operations in database is trackable.

Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.

Fred Brooks, The Mythical Man-Month: Essays on Software Engineering (1975, 1995), https://en.wikiquote.org/wiki/Fred_Brooks

With Fred Brooks' quote in mind, I approached again the issue of code first. Code could not come first. Data and datamodels came first. Code second. In fact, code comes third, because second place is given to apis, most notably the REST apis created with Dancer2.

Interfaces and Silos

Database schemas and system apis are both interfaces to data. They are the longest lasting parts of the system - and the ones that are the most difficult to change later. Code isn't. It can always be refactored and improved and tested against the unchanging interfaces.

If the interfaces are locked, especially the database schema, and we can be 99% sure that our data is always protected from a malfunctioning program, then it's time to give the programmer free hands to create the best code he can.

Furthermore, the future of our application is in constant change - like in many startups. Microservices is a natural way to extend this way of thinking. Different parts of the system become microservices and silos whose implementation code is their private part. This code can be quickly changed and it must have no connection to any other silos' code. This allows very radical changes if need be, such as web programming frameworks, math packages or even Perl interpreter version.

And what's best, none of the changes in the code threaten the stability of the whole system. Code quality becomes a matter of code reviews. Our backs (interfaces) covered, coding with Perl is fast and fun, because it just works (TM).

The Perl Way

There were several times when we were second guessing our decision to use Perl. This whole story happened in the course of one year's time. I consider myself lucky to have been given the chance to go through that whole mental process. I believe I understand Perl a lot better now - not perhaps as language but as a way of seeing software development and organizing development projects.

The first version was indeed only good for throwing away like Brooks writes in The Mythical Man-Month. We saved some parts and also some ideas from the second version. Speed is of the essence. The internal services for accessing database are mostly skipped and Dancer2 database plugin is used to fetch the data directly. Most action happens right in the same package where there the Dancer2 REST interface endpoints are defined because in most cases the data fetched from or written to database requires no additional handling. So there is no need to create additional layers, especially when the rigid database schema assures that fetched data is always sound (no nulls, no missing values or missing foreign fields). While quality control was earlier exercised only via code reviews, those are now complemented with api tests and rigid database schema modelling.

[Software engineering is the] establishment and use of sound engineering principles to obtain economically software that is reliable and works on real machines efficiently.

Friedrich Bauer (1972) "Software Engineering", In: Information Processing. p. 71