Data Management for Microservices
In this blog post, we'll be enhancing our RetailloopTube example application by incorporating two crucial components: file storage and a database. These additions are essential as they enable the storage of dynamic data produced and updated by the application, as well as a repository for assets accessed or uploaded by the application.
We'll start by integrating file storage into RetailloopTube, providing a dedicated space for video storage. This step is crucial for delineating the responsibilities within our application, particularly between streaming and video storage functions. This necessitates the introduction of another microservice into our architecture, marking the creation of our second microservice in the development process.
Following the setup of file storage, we'll move on to integrating a database into the application. The initial purpose of the database will be to store the paths to each video file. However, this is merely the beginning. The inclusion of a database opens up numerous possibilities for managing video metadata and addressing the data storage requirements of our microservices ecosystem.
By incorporating both a database server and an additional microservice, we're making significant progress in scaling our application. We started with constructing our first microservice in the first blog series, followed by deploying it using Docker in the second blog series. Now, we're expanding our application to support multiple containers, a step that necessitates the adoption of new tools and strategies. This blog post will guide you through the process of using Docker Compose to build and run your microservices application in a development environment, adding file storage, and incorporating a database to enhance the functionality and scalability of your application.
Developing Microservices with Docker Compose
A microservices application truly comes into its own when it scales beyond a single service. It's time for our application to grow, expanding its capabilities by incorporating additional containers. In this blog post, we'll leverage Docker Compose to transition our application into a more complex microservices architecture. By doing so, we'll demonstrate how to effectively manage and orchestrate multiple microservices, an essential step in developing robust, scalable applications. This move towards a multi-service architecture is crucial for realizing the full potential of microservices, enabling us to build more flexible, resilient, and scalable applications.
Why Docker compose?
Managing a multitude of containers during development can quickly become a complex task. As we venture further in our journey, future blog post will introduce Kubernetes as a solution for orchestrating containers in a production environment. Kubernetes, while powerful, is inherently complex and designed for multi-computer setups, necessitating at least one master and one node for operation. Simulating a Kubernetes environment on a single development workstation isn't straightforward. While Minikube offers a scaled-down Kubernetes experience suitable for local development, there exists a simpler alternative that you might already have installed—Docker Compose.
Docker Compose stands out for its ease of use in managing multiple microservices during the development phase. As an extension of Docker's capabilities, Docker Compose facilitates the construction, deployment, and management of multi-container applications, streamlining the development and testing processes. This tool becomes increasingly invaluable as we seek to efficiently boot and reboot our entire application stack during frequent development cycles and testing phases.
In previous blog posts, we explored methods for running microservices individually—either through separate terminal instances for each service or by deploying each container independently using Docker. While these approaches have been foundational in our microservices development journey, they become less practical as our application's complexity grows. The overhead associated with these methods diverts valuable time from actual development work, hindering our iterative progress, dampening productivity, and potentially diminishing our motivation.
Docker Compose, therefore, emerges as an essential tool in our toolkit, offering a more streamlined approach to managing the complexities of a microservices architecture during the development phase.
Creating Docker compose file
Create a docker-compose.yml file at the root of your project file, recall the dockerfile we created in the previous blog post, create the docker-compose.yml file where the dockerfile was created.
Steps
Create an Azure Storage Account => Storage account
In your docker-compose.yml file write the following
version: '3'
services:
db:
image: mongo:4.2.8
container_name: db
ports:
- "4000:27017"
restart: always
azure-storage:
image: azure-storage
build:
context: ./azure-storage
dockerfile: Dockerfile
container_name: video-storage
ports:
- "4001:80"
environment:
- PORT=80
- STORAGE_ACCOUNT_NAME=<your Azure storage account name here>
- STORAGE_ACCESS_KEY=<your Azure storage account key here>
restart: "no"
video-streaming:
image: video-streaming
build:
context: ./video-streaming
dockerfile: Dockerfile
container_name: video-streaming
ports:
- "4002:3000"
environment:
- PORT=80
- DBHOST=mongodb://db:27017
- DBNAME=video-streaming
- VIDEO_STORAGE_HOST=video-storage
- VIDEO_STORAGE_PORT=80
restart: "no"
In the latest iteration of our application, the video-streaming microservice now establishes a connection with the database. This integration introduces new environment variables, `DBHOST` and `DBNAME`, which are crucial for configuring how the microservice connects to the database.
A key aspect of our configuration, especially within the database container setup, is the port mapping strategy. We've mapped the MongoDB standard port of 27017 inside the container to port 4000 on our host operating system. This port mapping serves a dual purpose. Within the Docker environment, containers will use the default MongoDB port, 27017, for internal communications, adhering to the convention and ensuring consistency. Externally, on the host OS, the database is accessible through port 4000—an arbitrary choice that avoids conflicts with any MongoDB instance that might already be running on the host.
This configuration strategy is particularly advantageous for development purposes. It enables seamless interaction between our application and MongoDB using its standard port, while also offering the flexibility to directly access, query, and modify the database using external tools from our development workstation. This setup enhances development efficiency by facilitating direct interaction with the database, allowing for immediate testing and adjustments as needed.
Remember the index.js file at the beginning of the series, let’s update so we have the file connect to the mongoDB database:
const express = require("express");
const http = require("http");
const mongodb = require("mongodb");
const app = express();
const PORT = process.env.PORT;
const VIDEO_STORAGE_HOST = process.env.VIDEO_STORAGE_HOST;
const VIDEO_STORAGE_PORT = parseInt(process.env.VIDEO_STORAGE_PORT);
const DBHOST = process.env.DBHOST;
const DBNAME = process.env.DBNAME
function main() {
return mongodb.MongoClient.connect(DBHOST)
.then(client => {
const db = client.db(DBNAME);
const videosCollection = db.collection("videos");
app.get("/video", (req, res) => {
const videoId = new mongodb.ObjectID(req.query.id);
videosCollection.findOne({ _id: videoId })
.then(videoRecord => {
if (!videoRecord) {
res.sendStatus(404);
return;
}
const forwardRequest = http.request(
{
host: VIDEO_STORAGE_HOST,
port: VIDEO_STORAGE_PORT,
path:`/video?path=${videoRecord.videoPath}`,
method: 'GET',
headers: req.headers
},
forwardResponse => {
res.writeHeader(forwardResponse.statusCode,
➥ forwardResponse.headers);
forwardResponse.pipe(res);
}
);
req.pipe(forwardRequest);
})
.catch(err => {
console.error("Database query failed.");
console.error(err && err.stack || err);
res.sendStatus(500);
});
});
app.listen(port, () => {
console.log(`Microservice online.`);
});
});
}
main()
.then(() => console.log('Microservice online))
.catch(err => {
console.error("Microservice failed to start.");
console.error(err && err.stack || err);
)
In the revised version of our video-streaming microservices, the process for video retrieval has been significantly optimized. The code block above demonstrates how the system now queries the database using a video ID to locate the video's storage path. This path is then used to instruct the video-storage microservice to fetch the stored video. This mechanism ensures that HTTP requests for videos are efficiently directed to the appropriate storage location.
This update marks a pivotal shift from the previous approach, where video paths were hard-coded. Now, videos are identified through their database IDs, enhancing the system's flexibility and maintainability. While it might seem feasible to directly reference videos by their storage paths, this method is fundamentally flawed for several reasons.
Firstly, relying on storage paths to identify videos poses significant challenges if there's a need to reorganize the storage filesystem. Such changes could disrupt the system, especially since videos are interconnected with various databases that store metadata, recommendations, and viewing records. By identifying videos by their IDs, we significantly reduce the risk of these cascading issues, granting us the flexibility to adjust storage structures without impacting the overall system integrity.
Moreover, this ID-based approach simplifies the management of video locations, which can often involve complex paths. Keeping such internal details abstracted away from the external interfaces of our application is crucial for security and privacy. It prevents the exposure of information that could potentially reveal the internal configuration of our storage system, maintaining a tighter control over how our application's internal structures are perceived and interacted with externally.
Remember to install all necessary dependencies
Run the docker command by using ‘docker-compose up —build’
Integrating a Database Server for Production Environments
Up to this point, our focus has been on incorporating a database server within the development setup of our application. This approach suffices for initial stages, given that we have not yet explored deploying our application into a production environment. However, forthcoming discussions in next blog posts will delve into production deployment. Meanwhile, it's valuable to contemplate potential strategies for deploying a database server that caters to our production needs.
While Docker Compose facilitates the integration of a database server into our development workflow, transitioning to a production environment necessitates a different approach. For production scenarios, it is advisable to utilize a database that operates outside of the Kubernetes cluster. This strategy ensures the cluster remains stateless, allowing for its reconstruction at any moment without jeopardizing data integrity.
Upon establishing a Kubernetes cluster tailored for production, deploying a MongoDB database can be achieved with ease, mirroring the simplicity experienced with Docker Compose. This method, which we plan to explore in a future blog post, represents a straightforward path to introducing our database server into the production landscape.
Nonetheless, for long-term operational efficiency and security, it is prudent to maintain the production database as a separate entity from the cluster. Options include hosting it on an independent virtual machine or opting for a managed database service. This separation reinforces the stateless nature of the production cluster.
Choosing a managed database service brings added benefits, particularly in terms of security and maintenance. These services are dedicated to safeguarding and backing up data, relieving us from these responsibilities. The choice between managing the database in-house or leveraging an external service largely depends on the organization's size and resources. Larger companies may prefer internal management, whereas smaller entities and startups can significantly benefit from the external support provided by managed database services.
Choosing Between Database-Per-Microservice and Database-Per-Application Models
As we advance our application, having established a single database thus far, we're now poised to explore the creation of additional databases. The decision to name our initial database after the video-streaming microservice isn't coincidental but hints at a pivotal architectural principle we'll adhere to throughout our guide: the notion that each microservice should be paired with its own dedicated database. This approach mirrors the concept of data encapsulation in object-oriented programming, aiming to isolate data within its respective microservice.
The question then arises: Should we allocate a unique database to each microservice, or should we centralize our data into a single database for the entire application? The recommendation leans towards assigning an individual database to each microservice. While these databases may reside on the same server, segregating them ensures that each service operates autonomously. Integrating microservices through shared databases can lead to complex architectural challenges and scalability constraints.
This exclusivity in data management permits us to evolve our microservices independently, safeguarding the integrity of our data structures and minimizing the impact of changes across the application. Thoughtful design of REST APIs further enhances this isolation, preventing potential disruptions from spreading among services.
Although tempting, utilizing a shared database as a means for microservices to exchange data introduces fragility and limits scalability. Such integration practices are discouraged. However, exceptions may arise, prompting a reconsideration of these guidelines. It's essential to critically assess the necessity and implications of deviating from best practices before adopting such an approach.
Key Milestones Achieved:
- We've successfully integrated a database into our application, offering two distinct data management strategies: external cloud storage for files and a database for structured data.
- Utilizing Docker Compose, we've orchestrated an application comprising multiple containers, elevating our architecture to include two microservices and a dedicated database.
- The video-storage microservice abstracts our storage solution, facilitating future transitions between storage providers with minimal impact.
- By establishing a database server and dedicating a database to the video-streaming microservice, we adhere to the principle of one database per microservice, laying the groundwork for scalable expansion.
- We've initiated inter-microservice communication, as demonstrated by the interaction between the video-streaming and video-storage services, introducing the basics of microservice collaboration.
In the forthcoming blog posts we will delve deeper into microservice communication techniques and explore advanced Docker Compose functionalities, including automated live reloading for our microservices ecosystem.
**
Conclusion**
If you don’t want to wait for the whole series to end to get the knowledge, you can buy my ebook on gumroad => Microservices . To encourage to keep up the pace with writing this blog post, you can buy me coffee with the button below
Originally published on substack
Enjoyed this?
Subscribe to get new essays in your inbox.