Production-grade Configuration Management

Config or configuration management. If we go by the definition then we can say that config management is the systematic approach to organize or store or access or basically maintaining all the settings. This is the key word here. All the settings of our backend app. So you can think of this as a DNA of your application. It decides how your code runs in different different environments. When you say configuration management to most people then the first

impression that we get the first idea that hits our mind is storing some kind of database passwords the secure connection URL of our database or things like secure authentication keys secret keys for your JWT secrets etc etc or any API keys of external services like say your email delivery service storing API keys for external services like that. Those are the things that comes to our mind whenever we talk about config

management. But thinking like that misses a lot of scope in configuration management because that's like saying a car is just about the engine. Engine is definitely the most important part of a car but you're still missing out the 90% of the other features of a car. So similarly configuration management it it has a lot of other things starting from how your application starts up, how it connects to external services, how it behaves depending on different different

environments, what does it log, whether it logs or not and where it logs and where does it send all the details about your metrics, the performance metrics, your business metrics and things like which features are we enabling for this version of the deployment and which features are going to be disabled and for which users we are enabling which features etc. There is a lot of scope for config management and in this video

I want to touch upon what are the different things we care about what are the different things we think about whenever we talk about config management it's not just limited to your database URL or your secret keys it affects a lot of behaviors important behaviors of your backend application so definitely an important topic to discuss here so let's say you are building an e-commerce platform. You're building an e-commerce platform and if you are working on the

back end of an e-commerce platform then your configuration might include some of the things like what are the timeout details, what is the host, what is the port, what is the username, what is the password etc etc all the connection details of our database. It might also have your payment processor API keys. Let's say you are using stripe then you might have your stripe API keys there. Also your feature flags. What are the

feature flags that we are enabling for this deployment and what are the features that we are keeping disabled for this deployment. And these feature flags let's say they control we built a new checkout flow in our e-commerce platform and using a particular feature flag we can control that whether we are still using the old checkout flow or we are using the new checkout flow and we can also conditionally enable the new checkout flow for some users and the old checkout flow for some other users. So we use feature flags for these kinds of

instances. You might also have performance tuning parameters like what are we keeping our connection pool size connection pool that we use for our database connections. Also security settings like what is session timeouts whether it is 30 seconds 60 seconds and we can also store things like business rules. Business rules for an e-commerce. It might contain things like what is the maximum amount of order for a user and

each of these different configuration that we talked about starting from database connection details to payment processor API keys to feature flags to performance tuning parameters and security sessions security settings like session timeout settings and also business tools. All these different different configurations have different different characteristics. Some of them are sensitive. Some of them are supposed to be kept in secret. We are not supposed to leak them and they might cause a lot of damage. They might cause

a lot of loss. And some of them are not secretive but they impact they control how our application behaves while it is running. Also some of them might change frequently and some of them let's say they change once a month or once a quarter etc. And some of the configurations may stay the same for all our environments in our development environment in our staging environment and in our production environment and

some of the configs they'll be different depending on which environment that we are running our application. In all of this there is one challenge which is distributed systems. Most of our modern applications all our modern backends they don't run in isolation anymore. Most of them are now part of a complex distributed system. A system consisting of multiple services databases caches like radius etc or message cues or third party integrations or authentication or

emails and a lot of things. And all these different different integration points where we are connecting a separate service to our main back end. They all require configuration. Your back end obviously it needs to know how to connect to these services. But on top of that, it also needs to know how to handle failures, how to optimize performance and how to maintain security depending on different environment, depending on different settings and depending on different configs that we can control from different different sources that we will soon talk about.

And this is the reason the reason that most of our modern backends they run on distributed systems. This is why configuration management becomes an important part of your application. And if you don't have a systematic approach, if you don't have a dedicated pipeline, if you don't have a dedicated strategy to manage your config, you will end up with what we call as configuration chaos. Configuration chaos basically means that instead of controlling some configs from a centralized point, you

have hard-coded values scattered throughout your codebase. And you also have inconsistent behavior across different different environments. You also have security vulnerabilities from exposed secrets and debugging your application becomes a nightmare because you cannot really reproduce these issues because you don't have a centralized way of managing your config. So you don't really know what exactly happened and what particular config caused a particular issue, what particular config

caused a particular break in production. The stakes are pretty high for backend engineers these days since they handle the core business logic and data is pretty much the most important part of any platform any enterprise. So if you think about it, a misconfigured front end, it might show let's say a wrong feature, a wrong dialogue which it's it's not supposed to show or it might redirect to a wrong route. But a

misconfigured backend, it can cause a lot of damage. It can expose customer data. It can process payments incorrectly or it can bring down your entire platform if some important part of it is affected by it. Since backend systems run in diverse environments, different cloud providers, different onremises servers and containers and serverless functions and edge functions and all have their own requirement of different different configurations. Since we are talking about cloud

providers, let me tell you a little bit about the sponsor of this video which is Savala, a platform as a service provider and you can deploy your applications, your databases and your object storage, your deployment pipelines and you name it. If your team is paying per seat pricing in other platform as a service providers for your hosting needs, you are literally throwing your money away. I'll show you how you can save $300 per month while actually getting a

better infrastructure and support. So this is the math. If we take a look at this platforms like Render, they charge around $29 per month. And if let's say we talk about Railway, they charge around $20 per month. and platforms like Netifi, they want $19 per team member. So if you are a 5% dev team, that's around $145 to $290 monthly just for the privilege

of collaboration. And let's say your team is doubled. You have now 10 developers. Now you're looking at monthly $290 to $580 monthly just for the seat fees. You have other cloud hosting fees. Just for the seed fees, you are paying almost $500 per month before you even deploy a single app. If you can feel the frustration, then you can definitely check out Seala. They don't have any seatbased pricing. Seala is basically a Kinsta product. Kinsta,

these guys have been around for a long time. They are one of the most famous hosting platforms when we are talking about WordPress. And seala is their latest offering. Since they already had the infrastructure for all their hosting needs, they have created this product called seala. And the interesting thing about this platform is they don't have any seedbased pricing. Absolutely zero. Your entire team gets full access whether you have two developers or 200. But just because your costs have come

down, do not assume that you are going to get lesser quality. Seala uses GCP behind the scenes and the premier tier networking of GCP and it runs Kubernetes behind the scenes with 25 data centers around the world with 99% SLA. They also have an integrated cloudflare offering which has around 260 plus points of presence around the world offering you things like edge CDN DOS protection and

edge caching global load balancing and a lot of things that cloudfl natively brings. You get enterprise reliability without paying for the enterprisegrade pricing. Sealasa supports more than 20 programming languages through Nyx packs which is more resource efficient than traditional build packs. You get automatic deployments using GitHub, GitLab, Bitbucket, whatever version control system that you are using in your team and also zero downtime Kubernetes health checks throughout your applications. You get features like

hibernation pause for your development environments and horizontal scaling for all your production traffic spikes. But unlike most platforms, Seala has a human monitoring system. Their support team actually monitors your deployments. If you are facing a lot of errors and facing a lot of challenges in your deployments, they reach out to you to help you out and help your team out. That's not something you'll see in a lot of platforms. They also have a lot of compliance checks. You have sock 2, you have ISO, GDPR, CCPA, does not matter what your compliance needs are, they're

already there. So if that sounds interesting to you, check out Savala and you'll get $50 credits to try out different applications, database hosting, object storage, whatever your needs are. You can calculate your team's current seatbased cost and see how much you'll save as compared to other platforms. Your CFO will definitely love the budget relief and your developers will love the simplicity of the platform. So what are the different types of configuration that we are talking about here? Because not all

configurations are created equal and understanding the different types of configuration data is very important for choosing the storage mechanism, the security measures and the access patterns, who can access and what environments a particular configures should be accessible. So let's talk about what are the different types of configuration, what do they look like, where do they actually get used. The first one is our most common kind of config which is the application

settings. Application settings is the most common kind of config that you will see in most backend apps and this basically includes our log level and things like our port which port the backend application is going to run on our connection pool size our timeout values. timeout values like if you have an HTTP request then how long the server should wait for the request to get completed. So if you set a timeout value

for 60 seconds and let's say you initiate a request which generates an AI image, your back end generates an AI image and sends it back to your front end in base 64 format and the AI image generation takes let's say around on average 80 seconds. So if your server timeout is 60 seconds then your request is going to get dropped. you will get a status code of 504 timeout status. Similarly, log level in usually when we are developing our application on

development environment, we set the log level to debug so that we can actually see and we can actually differentiate between different different logs when we are creating the application when the application is running on our local environment. And when we go to production then we change the log level to info so that we don't clutter a lot of logs in our valuable production logs. Same port usually we run our back end app in local 8080 and depending on your

production setup whether you are using VPS or Kubernetes you might have different different ports. Same way connection pool size for our database connections or our database operations. Most backend apps use this technique called connection pooling. Connection pooling helps us optimize database connections. So we can also configure what is the maximum connection pool size that we want to have. So these are some of the common application level settings that you will see in pretty much all the different kinds of backends. Next up,

what do we have? Second, we have database. Database config. Now, database config basically includes all the details your application needs to connect to your database. This will include the host where your database is running on, the port of your database, the username of your database, the password and the name of your database. All these different different parameters

combined you can create a connection URL and that URL can be used when your application is running to connect to your database. It might also have other parameters things like timeouts how long your database query should run for before it times out. Moving on the third type of config that we can have is external services. Now external services can be a lot of different types. If you

are using emails then you might have an email provider like Mailchimp or something model like resend. Does not matter what provider that you use. You'll most probably have some kind of API key in your config that you will need to send the emails to initiate the client. Similarly, if you are using any kind of payment processor, let's say Stripe, then you might also have a Stripe API key. Or if you are using for authentication, if you are using something like clerk, then you'll also have a clerk API key. So all these

different different services that we call as external services, they also have their own configuration needs and you'll also need to cater to that. Then the next type we have is feature flags. Feature flags. Feature flags exist so that we can dynamically enable or disable particular features of our application. So as earlier I gave an example you are building an e-commerce platform and you have an checkout flow.

You have a checkout flow where we have all the order details here and here there's the payment details and proceed to payment. Something like this. And now you're building a new checkout flow. And for that you are using a different API and you don't want to roll out this new checkout flow for all your users. You want to do some kind of AB testing with it. You want to only enable this feature for a particular segment of your user depending on the location of the users. You let's say you only want to enable this feature for people who are from US

and you want to keep it disabled for people who are from India. If you have requirements like that, we use feature flags for dynamically enabling or disabling features in our application. So feature flags also need some kind of config management. So that's the fourth category. Apart from that we also have other types things like let's say infra config all your devops related configs and let's say your security config

your JWT secret your session secret anything related to security that your application is impacted from then we also have performance tuning performance tuning parameters so if your back end is built with something like go or go line then you can set different different environment variables like the maximum number of CPUs etc etc then also we have business rules different logic related rules that you want to enforce on application level but you want to

centralize that config so that's when we use business rule related config management so that's pretty much all about different types of conflict that you will face on your day-to-day life as a backend engineer the second important thing is different sources of config basically the storage of configs. If we have all these diverse needs of config management then definitely we'll also need to think about the storing of the

configs and the where do you and where you store your configurations it depends on a lot of parameters depending on security depending on speed and your environment. When we're talking about storage of configurations, then the most common kind that I have seen and most people probably have seen is the environment variables. And if you have worked with backends before, then envend

apps. is mostly common among NodeJS apps but we also have support for them in Python Golang does not matter what programming language that you're creating your backends in. This is the most common place where we store our configurations. If we are running in our local environments then we have a file called env. We use any library a library called env is very famous which basically takes all your environment

variables from env and it loads it in your operating systems environment. So this is basically a feature provided by your operating systems. This is not something you build but we have libraries which take these values from files and load it in our environment instead of us doing it manually with export. Similarly, if you're not in a local environment, let's say you are deploying in some kind of containerized environment using Kubernetes etc etc. So

those technologies also have support for loading your environment variables while they are deploying your application. So this is the workflow that happens when you deploy it in a cloud provider using environment variables. The deployment triggers at a particular stage it fetches all your environment variables from any kind of service let's say from vault which is from hashiorp or something like this parameter store or azure or google secret manager any cloud

provider which gives this feature of secrets management it fetches for it fetches from all these cloud providers and it loads that in your environment. And then when your application starts, it takes those environment variables and it functions whatever way it is supposed to function. That is the most common storage we have for our configurations. The second kind is files. Now in files we have different different formats. We store configurations either in JSON but

these days YAML is more famous for storing configurations. One of the major disadvantages of using JSON is you cannot add comments. JSON does not have support for comments. That's why most people will use YAML. There you can add your configuration and you can also add different different comments so that it's easier for your team to share knowledge. We also have a newer standard which is toml file. This is also pretty heavily used for managing configurations. So if we were to take a

look at a real world codebase. So this is some okay this is an example of AIA. This is a authentication provider technology. This is a Golang application. As you can see this have all these go files. So here you can see we have a configuration.yamel file and there are different different values. We have server we have log level. debug, storage, our notification related details, identity related details, our regulations, session related details and

all these things all of them are stored in a single YAML file. And if you were to search for config then in the results you can see we have config.yamel in a lot of different different places. This is from incubator answer is also another famous open source reposit from Apache and you can see they also have some fixed configurations for app settings that we just talked about application related settings like your which port your server should run on the databases

for local environments they are using SQLite and the swagger UI config UI config. So storing your configurations in a YAML file is pretty common in most of the opensource repositories. Moving on, we also have evalu. But we also have dedicated tools like console or etc. these cloudnative tools. We also have access to them. Depending on your use case, you can use any of them. Key value stores are pretty

lightweight and pretty simple to make use of as compared to other complicated and hierarchical data structures for config management. This acts mostly like your environment variables. Then we also have dedicated cloud providers cloud here majorly we have hashore vault. Then we have AWS parameter store Azure key wault all these different different providers. If let's say we take a look at Aship or Walt, you can explore all

these different different services which are dedicated for config management your secrets management that most companies use because we have deployment set up in a distributed manner. We have Kubernetes, we have autoscaling and we also sometimes make use of different different cloud providers at the same time for our scaling needs. So having our all our configurations in a centralized manner in a tool something like hashikov's vault which already is ready for all these different different environments. They already have dedicated support. They already have

dedicated docs and everything to make the integration as easy as possible. Using a external provider makes a lot of sense if you are running a big user traffic in production. But most of the time we also make use of hybrid strategies. Let's say we have a file where we load all our configs and construct a runtime application settings. And when we are in this loading phase, we might fetch our configs from different different places. In the first priority, we might fetch from AWS parameter store. Then we might

also have a config. DAML. Then we might also have some environment variables. So we can predecide that which of these should have the higher priority. And depending on the environment we can load those configs conditionally depending on the priority and the environment. So that's all about different different stoages for storing our configs. Now one question we have is why do we have different different configs depending on

the environment that we are running on. And the answer to that is pretty simple because each environment has a setup priorities that the other environment does not need or it should not have. For example, if we are running in a dev environment, whether it is your local in a dev environment, let's say in your local environment, you are running on local host, then our number one priority is developer productivity and our debugging capabilities. so that we can

work faster and we can catch issues faster. Similarly, if we are running in a test environment, we trigger a GitHub action and it prepares some kind of testing environment where we can run different different integration testings or unit testing or whatever testing setup that you have. Then in that kind of environment, our number one priority is automated validation and quality assurance. Similarly, staging environment. Staging environment is

basically a hosted version of our application that is as that works and functions as similar to our production environment as possible so that we have a clear idea what our production environment is going to work like and we can make changes fix issues and debug issues accordingly. So that is the priority of the staging environment. Similarly the last one in our production environment the number one priority is

reliability and security and performance. So we create our configs keeping these priorities in mind and that is the reason our configs are loaded in different ways. They are stored in different ways and they are different in their values depending on which environment that we are running in the application code. Of course, it remains the same, but depending on the config, the behavior of our application changes and that is exactly what we

want. Without changing the code of our application, which we might have to do if we are hard- coding things, but since we are centralizing our config management, we will never have to touch our application code. We can make changes in our config and that will reflect in the behavior of our application. So if we take an example of our database pool size in your development environment when you are running in your local host you can set your pool size max pool size for 10 and

it mostly works fine while you are testing your application while you are developing your application you don't really face any issues because local host these days most people have high-end devices and it can it has a lot capacity for these kinds of things. But on production, you have set the connection pool size to 50 because you have to cater to a large user base and you might get traffic spikes in different different times. So you have to be prepared for that. So you have set a higher database pool size in your

production environment. Similarly, in your staging environment, you have set the pool size to two because even though in staging environment, our number one priority is to mirror the functionality and the capabilities of our production environment as much as possible so that we can catch issues earlier. We can understand our production system better. But running a similar system which functions exactly like our production systems will cost the same amount when it comes to our cloud costs and that's

not something a lot of teams want. We want to save as much money as possible when we are talking about cloud costs. So in our development environment our another set of priority is minimizing our cloud costs. So database pool size since we already know that a staging environment is usually is used by our developers or our testers. So we can take some decisions like keeping our database pool size to minimum and we can accommodate some kind of delay there but

we'll save a lot of money in our cloud costs. And the last thing that we have to talk about when we are talking about config management is security. These are pretty obvious points, but just for the sake of covering all the scope, I'll mention these. Never hardcode your secrets. If you have secrets like your production database URL, the API key of your clerk service or your email delivery service or your payment

processing service, you should never hardcode those secrets in your codebase, right? It's pretty obvious. I don't even have to explain why you should not. Second thing is if it's possible always try to go for a cloud secrets management service something like as I showed earlier hashikorp vault or AWS parameter store or Azure key wault or Google secret manager. It's always a good idea to overengineer when it comes to

security of your backend application. These services they already handle things like encrypting all your configs when they are stored in their storage. It does not matter where they store your config but whenever they store it they encrypt it and then they store it somewhere in any kind of database or any kind of storage. Similarly, when they are in transition, when you make an API call and you fetch all the configs, they send you the encrypted one and with your

private key which is stored in your infra or your GitHub action or your repository secret or your Kubernetes environment there they are decoded and then you use them. So they are encrypted when they are stored and they are also encrypted when they are transferred from one point to another when you are fetching them. Now these are essential security features and when you use a cloud provider like a vault or as parameter store all these things are already taken care of and you don't have to think about these. Third thing is

access control. If you have a fairly large developers team you should take some time to strategize who should have access to all your configs following the rule of least privilege. Developers who are working on front end, they should only have access to all the configs that they need for functioning things like the API URL of the back end and different different API keys for front-

end integrations etc. And if you have backend engineer, they should only have access to your databases, your radius instance, your elastic search, etc., etc. But something like configs for accessing your cloud instances, your EC2 instances only your DevOps team should have access to those. So following the least privilege principle, you should strategize your access control and you should also have things like rotation periodically. You should rotate all your configs, all your API key secrets, all your JWT secrets and things like these

so that they are not very prone to leakage. And the last thing that I want to mention here is validation. Most of the times that I have personally seen, we don't always validate our environment variables or whatever source our configs are coming from. Let's say if we are using env we load all the variables in our environment and we just access it using process env dot whatever the name of the

variable is. But one of the key strategies that can help decrease a lot of headache is when you are loading your config which is usually the start before your application starts after it is deployed you should validate all the configs does not matter whatever source they are coming from if they are coming from they are coming from AWS parameter store if they are coming from a file let's say config.l ML file always validate them using a proper validation

library. If you have a TypeScript back end then you can use ZOD. If you have Go, you can use Go validator playground library, any kind of validation library that your programming language, your framework offers and properly validate all the variables depending on which ones are mandatory to provide, which ones are optional and you're setting the default in your application code and thinking about all these different scenarios. Always validate your configurations. I have learned this the hard way. It can cause a lot of damage. It can cause a lot of confusion in your

production environment because it's not very easy to spot that you have a mandatory environment variable requirement but you were not able to provide it and now your production system is either breaking or behaving in a strange way because it does not have access to a particular environment variable. If you were to take one thing from this video, this is the most important part. Always validate your conflicts. Does not matter where they

are coming from. on a smile. The