Skip to main content

Making the switch from synchronous, tightly coupled to asynchronous, loosely coupled

· 5 min read
Nitesh

Last year, I developed SAAS (Software as a Service) for an internal team, allowing managers and team leaders to create forms from straightforward user interfaces and distribute them throughout the team for a variety of uses. Along with the form builder service, we also built an integration feature that enables leaders to export responses from our SAAS Tool to applications like Gmail, Google Calendar, and custom Webhook, enabling them to transport responses to the locations they want for report creation. The issue now is that the employee filling out the form is complaining that the platform is taking too long to submit the response.

Problem​

Therefore, the overly slow form submission process is the problem. A new task is created. let's see what i learn from this and how I resolved this issue.

GIF of the form builder tool is shown below​

Here is a straightforward GIF that illustrates how leaders can create forms utilising user interfaces.

form_builder

UI for Integrating form with third party tools (Gmail, Google Calendar,Webhooks)​

The user interface (UI) below demonstrates how the leader can connect a certain form with third party tools. here connect means whenever a new response come it will also exported to all the third party tools.  

notification

Brainstorm​

How and what data do I gather?​

  • I start by opening the form's frontend, where employees are filling it out.
  • I looked at the network tab and noticed that the backend Rest api's average response time was longer than usual.
  • Let's have a look at the Rest api's design and sequence diagram, which we are using to send/submit the response.

Submit response API's UML sequence diagram is shown below​

synchronous

Let's choose a flaw in this design​

  • first problem is our api design is not following the single resposibility principle which says one function/api should perform only single action. that means submit response api is only resposibile for saving the response in database.
  • second is the coupling effect is very high (means our api is highly dependent on third party tools) which causes 2 issues:
    • what if in future we have to add more number of third party tools for integration then we need to update this api and rebuild the service which is not the right way.
    • what if we need to remove some tools in future then we need to again iterate the same process which we not want to.
  • third and most important problem is synchronous nature which means we are waiting for each and every third party module to complete because of which response time of api is too slow.

Solution​

Let's see what we need to do in order to solve this issue​

  • so first we need to remove the tasks/functions of exporting data to third party tool from api. and put those tasks/functions in another module/service.
  • now we have 2 module/service one is responsible for saving the response in db and second is responsible for exporting that response to third party tools.  
  • but now the problem is how we can do the communication between these 2 modules to successfully export the data. let's see what we can do:
    • so we need a system which helps us to do the communication part between these 2 modules.
    • There are two basic communication patterns that services/modules can use to communicate with each other
      • first is synchronous using the request response pattern
      • second is asynchronous using message queues

Message Queues for communication between modules​

we need asynchronous communication between 2 modules. so first module is producer of the meassge/fornm response payload, second is the consumer.

so what message queue will do is it will act as a broker between producer and consumer. so producer will just put the message inside queue and consumer is polling message from queue.

this is how communication is asynchronous that producer(in our case submit response api) will not wait for the response, it will immediately push the message/form response payload to the queue.on the other hand consumer will take the message from queue and export that message/form response payload to third party tools.

Basic Message Queue Working​

Updated UML Sequence Diagram which depicts the curent System​

sequence_diagram_pub_sub

Reliability Talk​

it is important that your system should be fault taulerant and reliable which means what if your system fails. how you will deal with the situation when your SAAS tool can't export the data to a third-party tool due to system failure.

failure situations can be​
  • so producer pushed the message in the queue but system fails and consumer can't able to consume the message in that case our data is lost and hence not exported to third party tool.  
  • what if before pushing the message to the queue our producer fails then also data lost.
  • What if one of your third-party tools is down at the time of sending the message? so that third-party tool can't get that message. so data lost

How do we achieve reliability?​

  • there should be a system of acknowledgement(but keep in mind it should be asynchronous), so when the consumer consumes the message, it should notify the system that the message has been consumed.
  • if no acknowlegment is there we need to repush/retry the message to the queue.so that it will not be lost.
  • make your consumers idempotent(means prevent the same data to be submitted twice) . You can read the story related to idempotency here → story of idempotency

reliable_queue