Experience report on optimizing software architecture

Lately I've been thinking a lot about software architecture decisions and sustainable programming. In fact, that's what I've been doing for the last few years. I re-evaluated and rethought many decisions and constantly learned new things.

Even now, the learning process is far from complete. New projects and new people always bring new impulses and opportunities. But I would like to take the opportunity to try to describe what “a software project” actually is for me.

And that's not that easy, because I want to make it as interesting as possible for you as a developer, or as understandable as possible for you as a non-developer. That's why I decided to do this using a concrete example.

The example

Let's dream: let's say we have a company in the car insurance sector - a startup or an already established company. Since we keep up with the times, we have decided that we want to build or restructure our company in a data-driven manner.

Here is the first insert: data-driven – what does that actually mean? I don't think there is a clear definition. As entrepreneurs, we want to act as profitably as possible. This means a realistic risk assessment for our customers. At the same time, we want as many customers as possible, which means we have to have an appealing product. We obtain a reasonable risk assessment with the most accurate and up-to-date data from our customers. Of course, this includes master data such as address, age, place of residence and professional group. At the same time, behavioral data also counts here: do customers stick to the speed limit, do they drive daily or only occasionally, how long are the distances, are there an above-average number of accidents on the daily commute, and so on. This means that there is largely constant data in the form of master data and changing or constantly being added data. Capturing and utilizing these is a work of art. We are not talking about a momentary collection of data, but rather a historical collection including, if necessary, a real-time evaluation of the current data.

But butter to the fish: how do we actually get this really helpful data?

Since, as mentioned before, we are moving with the times, of course with an app. Customers install the app on their mobile phone and it is set every time they travel. The app then transmits the current GPS data and other sensor data from the cell phone to us in real time. This gives us all the information we need to evaluate driving behavior and determine a risk score.

But what does real time actually mean?

This is a term that is often used casually. Theoretically, we want to be able to determine retroactively at any point in time when our customers are. Technically implemented, you can imagine that we write every GPS point, including the time, into a continuous file. We then refer to real time as the smallest possible time interval in which GPS data is placed in this file, which we are satisfied with. This can be every nanosecond, millisecond, second, minute or even hour - although the concept of real time can often be questioned, especially in large time intervals.

Now, of course, another question arises: What do we do with the collected data?

The answer is one of the sexiest answers there is: machine learning. Specifically, customer clustering and cluster-related time series analysis come to mind. But these are just two possibilities out of many. The fact is: data is there to be evaluated. And that is exactly what needs to be done.

Now that the framework is clear and we have understood the product, let's get down to implementing it.

Homepage

I didn't mention them in the points above, but they shouldn't be forgotten under any circumstances. How else are customers supposed to find and contact us online? Instagram, Twitter and Co. are certainly interesting and relevant, but there is no way around a homepage.

The homepage has nothing to do with the product itself. It is used to attract customers and, if necessary, to handle queries. This can be implemented in any way. PHP and WordPress are probably the most common variants. Personally, I'm not a big fan of WordPress and tend to prefer Python solutions - in this specific case, probably Python Django.

The homepage should have a landing page & customer funnel as well as a user area. At best, dashboards about past driving behavior are displayed in the user area or customer support features are made available.

If you use WordPress, you really have to want it. I don't want to

Mobile app

This is where the real monster lies. The discussion about whether hybrid or native is quickly answered: you have to actively access the smartphone's sensors for the app. This is usually best done with a native app. In the worst case, you have to develop the same app in different programming languages.

But much more interesting than the app is how the data is stored. Because we need this data for evaluation in real time on our server.

The first consideration here is of course to provide a simple Rest API and database. As shown as an example in the diagram.

A first plan: A mobile device sends data to a server. These are written into a database and evaluated in real time by an AI. This initially does nothing with the data.

Before this is set in stone, it makes sense to take a quick breather - it's not that easy!

A big problem is synchronous communication. Data is sent from the cell phone to a server with a postal request. Before the cell phone can continue, it has to wait for the server to respond. But this only occurs as soon as the data record is successfully written into a database and after it has been checked whether there is an anomaly.

This is all suboptimal. It would be nicer if the anomaly detection reported proactively and this process was decoupled from sending the data.

In addition, a GPS signal may have to be sent constantly (e.g. to determine the current speed and compare it with the speed limits in the route section), or the cell phone sends information about G-forces and their directions to the server so that braking behavior or driving behavior in curves can be evaluated. Maybe more data will be added. But the fact is: that is a very large data throughput.

And suddenly you can consider whether a normal Rest API is still suitable or whether you would rather provide a long-lasting connection for constant data exchange using web sockets.

At the same time, this is more about data with a clear time dependency - in other words: time series. Databases such as Prometheus, InfluxDB or, if you are using AWS, TimescaleDB are ideal here.

A bottleneck that arises here is the high frequency of writing to a database. There are actually always problems. One consideration here is to decouple the web server from the writing process into the database, e.g. with a data streaming solution like Apache Kafka in the middle.

Second draft: Instead of a simple HTTP interface, Websockets are used, another web protocol, which allows a continuous data stream to be maintained. At the same time, you don't immediately write to a database, but rather add a tool - in this case Kafka - in the middle. This means that individual processes are decoupled. The overall process is much more complex and may take longer, but the individual components are faster and easier to scale, which can be particularly relevant in the API server.

And hey presto – away from the run-of-the-mill Rest API MySQL structure, towards an exciting, complex, customizable, scalable solution to handle the same problem. And one can justifiably ask oneself why one should voluntarily decide on a solution that is as complex as possible – why seemingly shoot oneself in the foot?

Unfortunately I can only say soberly: comparing this will be a separate blog post. I'm torn here. On sunny days I choose the first solution, on cloudy days I choose the second solution.

By the way, there are other ways to implement such a project apart from microservices. Specifically, Akka even comes to mind here. This enables event-based, distributed and persistent data processing but remains largely monolithic (unless you want something else). I think Akka is a very interesting insider tip for certain cases. That would also be a separate blog post.

Data Science & Machine Learning

Now that the architecture has been roughly thought about, we move on to the really exciting part.

Essentially, different types of data science are compared:

• Implementation of preventive measures e.g. anomaly detection
• implementation for time-delayed analysis purposes
• Corporate Abalysis Purposes

There are countless approaches here. In this example, an Apache Spark approach was chosen.

Spark has the wonderful property of being able to distribute processes across different machines. Spark can also wonderfully combine different data sources and give a data scientist the opportunity to work with different data sources in the same way.

In this specific case, Spark is used both as a pipeline (writing directly into a database accessible to the user in order to retrieve later analyzes of their own driving behavior), as well as for training models for anomaly detection, as well as for visualizing data as a purely company-internal requirement.

A real all-purpose weapon.

What's next here?

Well, so far we have only talked about implementation without actually implementing it. Although...that's not entirely true. Take a look here: FunWithMicroservices

Here you will find a concrete implementation of our project in a small style by me.

But apart from that, we are at a point where we have to consider: How much, how good and how expensive does it actually have to be?

And here it just has to be clear: projects like this are always a huge tradeoff. The concrete implementation also includes appropriate maintenance, further development and, above all, so-called monitoring or testing.

I have often experienced and heard that customers should have implemented a prototype. Prototype often means a test version of software for as small a group of users as possible. Often the provisionally small circle of users grows unexpectedly quickly. Or in other words: the prototype is not only used for test purposes as agreed, but also productively for real operations.

That is exactly what is legitimate. Because on the surface there is a usable product that works exactly as you would expect.

But what often happens is that feature development and bug fixes mix in large quantities and bugs accumulate with new features, or that new ailments from users constantly arise.

And to make this clear: an application is always more than what you see and use. In order to keep an application running, you have to make it observable as a developer. This includes seeing as simply as possible what the load distribution looks like on the servers. If we use Kafka, we would definitely like to see how many measured values ​​have not yet been processed (this is called the consumer gap. If the gap increases, it means that something is wrong with our application). In the case of our API server, we may want to see whether our instances are balanced properly, or we generally want to see whether all of our servers are currently “healthy”, because in the end everything runs on real hardware that can fail at less than 1%.

At the same time, we want to make sure that applications work as expected by, for example, storing logs permanently and centrally.

Of course, we also want to be able to develop as quickly and efficiently as possible, which often means that the entire deployment process has to be automated.

At the same time, it must also be ensured that the features work permanently - even after changes. Tests are written for this purpose.

And all of this is just the tip of the iceberg. But all of that is somehow part of the software. And everything is associated with a very indirect and hypothetical benefit for the product as well as very concrete costs. For this reason, the question must always be asked: What do we actually need from this?

But yes - there is another huge area hidden here about which you can of course write your very own blog article.