The BillRun is built on MongoDB, which is the open-source, document database, popular among developers and IT professionals due to its agile and scalable approach. MongoDB is also known as the leading NoSQL database technology.
Recently, we've received a lot of questions about how we implemented a billing system without a database which supports transactions nor row locks.
In this article we will try to explain how to verify integrity and concurrency using a MongoDB database. In addition, this article dispels the myth about MongoDB not being able to deal with ACID (atomic, consistent, isolated and durable) operations.
BillRun entities background:
- Customer entity is account and it has multiple subscribers.
- Line is the raw record data
- Queue - An ordered collection of lines to be calculated (type, volume, rate, price, etc)
- Billrun - the aggregated data with grouped by main factors (such as usage type), and divided per account's subscribers.
- Plan - the subscription type of the subscriber. This affects the rate and cost of the subscriber's usage.
Calculators & Aggregators
One of the main components of BillRun are the calculators. Those tiny engines take care of pricing, rating and much more. A calculator can calculate the usage from its CDR (data record), and convert it to cost by identifying the line rate.
Another main component is the aggregator which aggregates the usage of each customer. This aggregation is done on run time. Therefore you could know the total costs very fast and on real time.
So how we insure information reliability?
The first level is by isolating the sequence of operations, therefore we use a queue and lock a line once a component uses it. No two processes can deal with the same line simultaneously.
Customer calculator
Integrity - if a subscriber could not be identified or that his plan could not be identified then he is returned to the queue for retrying.
If, at any stage, the process crashes, then the processed lines would be calculated from the beginning after the queue orphaned time has reached.
Concurrency - the customer calculator creates for every subscriber his usages document (MongoDB record). This document is used to detect when a subscriber exceeds his plan usages. If two customer calculators try to create the same document (same subscriber id & time period), only one of them will succeed. The query looks for an existing usages document, and creates one only if it did not find any. This is achieved using upsert option of MongoDB combined with the setOnInsert operator.
Customer pricing calculator
Integrity - when updating the usages document, the pricing info (price & over/out plan status) is saved as well as a backup. If the process is halted (e.g. because of a system crash) right after updating the usages document but before incrementing the subscriber's costs, on recovery it will continue with the saved calculated pricing from before the crash.
A check ensures that a line could not be added twice to a billrun document.
Concurrency - multiple pricing calculators could check & update the same usages document concurrently. A copy of the usages document is saved to the memory and the price of the line is calculated by it and by the subscriber's plan. Then the usage and the calculated price of the line are saved back to the document. However, if during the saving process it is observed that the current usage counters of the subscriber have changed since the last time they had been sampled, then the price of the line is re-calculated in order to take the updated usages into account.
This can occur when two calculators deal with lines of the same usage type (e.g. both sms lines) from the same subscriber at the same time. It is promised not to get into an infinite loop here since one calculator is always guaranteed to succeed in updaing the usages document. Moreover, such a scenario is extremely rare because of the expected vast number of subscribers.
As for the various billrun counters: if two calculators try to update the same account's billrun document simultaneously they would both succeed as the update and increment of the different billrun counters is done using the MongoDB findAndModify which provides atomicity.
Customer Aggregator
Integrity - a flat line (fee) for a subscriber for a given month could not be duplicated (in code & db level), and its update to billrun is consistent as in the previous stage. In addition, updates to a closed billrun are prevented in code level.
Concurrency - it is harmless to try to aggregate on the same accounts from different servers. The system will remain in a consistent state after doing so and the aggregation would occur only once per account: As mentioned above, flat lines could not be duplicated and the billrun update query includes a check to prevent double insertions.
Concurrency between calculators and aggregators
Consider the case when a pricing calculator and a customer aggregator work simultaneously. The calculator calculates some account’s lines and saves them to their billrun. In between, the aggreagtor closes the matching billrun of all these lines. The pricing calculator would then save the next line to the next open billrun (the runtime billrun).
More info
You can found more info in the MongoDB knowledge base:
