My first microservice solution - great mentorship, companionship and an unforgettably enjoyable development adventure

During my training period following my intership at AllPhi I was guided by Angelo Dejaeghere - who challenged me to create a fleet management application with a microservice architecture.

Meanwhile Brendan Daley, the developer I collaborated with during the intership, was challenged with the same mission, we both went our own way but shared knowledge, links and our professional opinions about - among other things - frameworks, methodologies, (cost-benefit) analyses - in all, a very valuable companion to work with.

Angelo guided us through our journey but allowed us to make our own decisions - most of the time - he identified our individual boundaries and adapted accordingly. He knew when to provide an alternative path, at the right time, every time.

Initially I wanted to go the orchestrator route using the chinese DTM library (Distributed Transactions Framework), which uses compensating transactions - among other peculiar and interesting features.
Big chinese tech giants such as Tencent and Bytedance incorporated this framework in their stack.

🔗 DTM @ construct0-forks

🔗 Adoption by big tech

Guided towards a choreography approach to model the design with, it was initially perceived as being challenging as it appeared more prone to errors.

The challenge was to switch mindset, learn more about the aforementioned approach - in order to streamline the process of formulating a solution which would allow events and their state to be managed, with respect to a Service's desire to be autonomous.

While perusing further through the microservice ecosystem my attention was drawn towards MassTransit, a feature-rich distributed application framework - but I was put off by the library's scope and it's many facets.

Some of the aforementioned facets already exist as separate nuget packages. Off the top of my head, my reasoning at the time was that this broad scope meant diminishing:
- separation of concerns
- modularity
- dedication & thoroughness - "indepthness" to one particular domain (generalization vs specialization)

Some of its conceptual implementations are simplified however, decreasing the likelihood of error. Foregoing functionality for simplicity's sake (along with determining whether manually completing a task or automating it; and thus determining development time required), are, in my opinion, the hardest challenges when it comes to developing a sustainable software solution.

Back to the project, it was overkill to apply this library in this scenario, it was also a tad intimidating to master its entirety for the sake of a first introduction to microservice architectures covers.

🧐 Perhaps at a later time I'll give it a go, the videos by it's lead developer on YouTube are particularly interesting and well composed.

Scope of the library
🔗 MassTransit @ construct0-forks

Going in conversation with Brendan, we both agreed that the EasyNetQ library, which offers a higher abstraction than the lower-level RabbitMQ client library, was the right choice. EasyNetQ acts as an abstraction layer on top of RabbitMQ and uses the RabbitMQ client library internally (only applies to .NET Standard 2.0).

📑 On the topic of abstraction
"We have used too many levels of abstractions and now the future looks bleak"
@ construct0/ReferentialResources
(src: unixsheihk.com)

 

🔗 A decade of dedication and passion by its architects and developers

EasyNetQ embraces KISS and doesn't leave the wide array of knobs and settings wide open for the developer to be overwhelmed by, ideal for a first application, configurable on-demand yet also not permanently opted in to heavy configurational overhead without alternative avenue.

While offering a higher abstraction, it also offers the possibility to modify settings & configurations, behaviors, and more (by writing extension methods or by forking the repository yourself) - where needed, offering the ideal design and development experience.

Still some work was required to make it the right fit for what I had in mind.

The most essential part of the application, an event broker intermediary BackgroundService dubbed EventBrokerIntermediary and an exemplar of another Service's Event Consumer, dubbed EventConsumer, which receives and processes the event, the EventRepository was chosen to be omitted as its purpose should be self explanatory.

When a service starts the EventBrokerIntermediary BackgroundService, an SQLite database file is stored within that Service, by the EventRepository, and used as its single source of truth, increasing separation of concern when it comes to event related storage.

Entity Framework Core, Dapper, SQLite as the data storage stack.

Other classes within the EventLibrary act as support and are not shown, nor discussed, except for the EventBrokerIntermediaryRetryHandler.

The EventBrokerIntermediary can be configured to:

• prune TrackedEvents with event state Completed
• determine exponential vs linear wait periods for retries
• max amount of retries

Let's take a look at the majority of the EventBrokerIntermediary's inner workings, its event loop, and how a ServiceEvent is processed by a Service's Consumer.

The EventBrokerIntermediary and an exemplar Consumer of a different Service
(partial diagram, full diagram below)
(🔗 full size diagram)

Each Service refers to a central EventLibrary where the EventBrokerIntermediary, EventRepository and their supportive lemmings are located.

Events namespace

Other essentials within the EventLibrary include the ServiceEvent, TrackedEvent and the events of the Service which it is intended to originate from. An additional Any folder is for events such as the FailedToProcessEvent which can be used by any Service.

The EventLibrary has the Services as its dependants - every Service is aware of the entire event model, but not the internal model of other Services.

But why would you use microservices at all if you're working in a monolithic application?

The reasons being:

• for the sake of learning, a monolith application with different projects was chosen, one for each Service, one for the EventLibrary, while adhering to the separation principle
• projects may be transferred to a separate solution if deemed necessary, while preserving the ability to reference the EventLibrary
• the EventLibrary could be made accessible for other (common) programming languages by using subprocess calls or protobuf files and RPC server.

Centralization of events and event related logic does not imply a failure, nor does combining Services within a solution.

💡 Google uses one repository for all their projects, with custom tooling as an aid 

Let's take a look at the diagram in its entirety, as I did not want to immediately overwhelm the reader with the size of the diagram.

Worth including in the diagram was how retrying is performed. That is the task of the EventBrokerIntermediaryRetryHandler, it is located in a separate class to decrease the size and complexity of the EventBrokerIntermediary itself (in the same vein of the EventRepository).

(🔗 full size diagram)

It should be pretty self explanatory how the EventBrokerIntermediaryRetryHandler operates, worthy of mentioning is that it is stateful and each TrackedEvent has its own handler instance.

On first retry, the retry is immediately performed, otherwise exponential or linear sleep times of the thread apply.

The EventBrokerIntermediary publish function is supposed to be called using the fire and forget method of calling async functions in C#.
If the service requires an update on the ServiceEvent's state, it is expected to use the EventRepository or a derivative thereof.

💡 Use a discard variable followed by a function call to perform fire and forget in C#, in this case the originating line of code responsible for creating the event and calling the TryPublish function of the EventBrokerIntermediary (step 2) is expected to adhere to this technique to avoid stalling - which could be significant if your ServiceRequestConfig's parameter values are misaligned.

A worthwhile addition could be an additional wrapping function for TryPublish, performing the fire-and-forget within the EventBrokerIntermediary - to lighten the burden imposed on developers.

When desired, a dedicated function within the EventBrokerIntermediary acting as a passthrough could be considered, if the EventRepository class is intended to be shielded (internal [class] modifier) from a Service's reach, as to avoid introducing programmatic error(s) in the future.

While the SQLite database being stored in the EventLibrary enables the same feat, it is not low hanging fruit anymore and it would require duplication of the DbContext and DapperContext, which should be marked as internal as well, in order to minimize the chance of introducing programmatic errors.

Packaging the EventLibrary as a nuget package, with correct usage of the internal [class] modifier, the chance of occuring would be possible with considerable effort and deviation only.

Conclusion
It's to-date the solution I am most proud of - and which I recreated after my contract ended to learn more and in order to improve it even further.

Thanks to the great companionship, and professionality which, Angelo & Brendan, accompanied by other valuable consultants and managers at AllPhi provided.

Thanks for taking the time to read about my experience.
Consider joining AllPhi as a .NET software developer.

~ Benjamin

Note
Repositories are forked to the construct0-forks organisation on GitHub and linked that way to represent the repository as it is on the date of publication.

While it is possible to backtrack to the appropriate commit hash height to view the state of the repository back then, this is a good way of removing friction for the reader.

For the most recent, up-to-date version, refer to the upstream repository.

The first reference of the library refers to the documentation.

For scientific papers or important articles, a web.archive.org link is provided when possible, or as an alternative, stored on construct0's ReferentialResources repository.

Disclaimer: with respect to the transfer of information and guidance, and despite the topics being in the public domain, any information contained within this article reflects how the codebase of my second iteration is composed for which development was initiated after my responsibilities and obligations allowed me to construct and subsequently share it.
Source code will not be provided. Details about work performed for clients will not be shared, or severely redacted in its description as to not lead to identification or and of a client's information. LinkedIn is rife with similar diagrams and concepts, this is merely an experience, accompanied by some knowledge, made possible by AllPhi's vision.