<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[The Cloud Is The Starting Point ☁️]]></title><description><![CDATA[The Cloud Is The Starting Point ☁️]]></description><link>https://folaav.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 17:00:55 GMT</lastBuildDate><atom:link href="https://folaav.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[My First End-to-End Secure AWS Backup Pipeline: Building, Testing and Learning Along the Way]]></title><description><![CDATA[Follow my journey building an automated AWS backup pipeline using EC2, S3, IAM, Lambda and DynamoDB. Explore architecture, security decisions, testing]]></description><link>https://folaav.hashnode.dev/aws-automated-secure-backup-pipeline</link><guid isPermaLink="true">https://folaav.hashnode.dev/aws-automated-secure-backup-pipeline</guid><category><![CDATA[AWS]]></category><category><![CDATA[lambda]]></category><category><![CDATA[aws-pipeline]]></category><category><![CDATA[aws backup]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Cloud Computing]]></category><category><![CDATA[Cloud Engineering ]]></category><dc:creator><![CDATA[Fola A.]]></dc:creator><pubDate>Sat, 21 Nov 2026 19:24:30 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/5ad805b8-9288-4ddc-88d9-76be5b51fa82.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>Introduction</h2>
<p>Going into this project, I had some familiarity with AWS, but this was an opportunity to see how its services could be combined to solve an actual infrastructure problem. The goal was to create a backup process that could run independently, without exposing the application server to the public internet. Beyond storing files, I also had to consider how to control access to those backups and keep track of every upload. This meant thinking about more than just storage; I needed to build a system where networking, security and automation all worked together.</p>
<p>This meant that I couldn't simply create an EC2 instance, install the AWS CLI and upload files to S3. I had to think about networking, access permissions, automation and monitoring, and how all these components would work together.</p>
<p>In this article, I'll walk through how I approached the project, the AWS services I used, some of the challenges I encountered and what I learned from the experience.</p>
<h2>Understanding the architecture before building</h2>
<p>Before creating anything in AWS, I needed to understand how the different components would communicate.</p>
<p>I started with a Virtual Private Cloud (VPC), using the CIDR block 10.0.0.0/16. Within the VPC, I created two subnets: a public subnet and a private subnet.</p>
<p>The public subnet would host a bastion server, while the actual application server would run in the private subnet. The reason for this separation was security. The application server needed to remain directly inaccessible from the internet while still having a controlled way to connect to it for administration.</p>
<p>I configured an Internet Gateway for the public subnet and separate route tables for the two subnets. I also created security groups to control SSH access. The bastion host only accepted connections from my own IP address, while the application server only accepted SSH connections from the bastion host.</p>
<p>This was one of the first things that helped me understand that cloud infrastructure isn't just about launching virtual machines. Where a server sits in a network and who can communicate with it are equally important.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/34f69b30-bff8-4331-bc6c-5e33e38496ac.png" alt="VPC resource map showing subnets and route tables" style="display:block;margin:0 auto" />

<h2>Getting the application server ready</h2>
<p>With the network in place, I launched two EC2 instances: the bastion host and the application server. The application server was deliberately launched without a public IP address. I accessed it through the bastion host, which acted as an intermediary between my computer and the private server.</p>
<p>One requirement that initially needed some thought was installing the necessary software on the private instance. Since it had no direct internet access, I had to temporarily introduce a NAT Gateway, which allowed the instance to download packages and install the AWS CLI.</p>
<p>I didn't want to leave the NAT Gateway running permanently because of the additional cost. The plan was to remove it after installation and use an S3 Gateway VPC Endpoint for communication with S3.</p>
<p>This distinction became important later when testing whether the backup process could continue without general internet access.</p>
<h2>Configuring S3 and managing permissions with IAM</h2>
<p>Next, I created a private S3 bucket to store the backup archives. Block Public Access remained enabled, and I kept the default encryption settings. However, creating a bucket was only part of the solution. I also needed to decide what the application server should be allowed to do with it.</p>
<p>For this, I used AWS Identity and Access Management (IAM). Rather than storing AWS access keys on the EC2 instance, I created an IAM role and attached it to the application server.</p>
<p>I created a custom policy that allowed the s3:PutObject action only on objects inside the bucket's backups/ folder. This meant the server could upload new backups but did not have permission to delete existing ones.</p>
<p>This was an important part of the project because the requirement wasn't simply to upload files. It was to ensure that existing backups were protected from deletion by the application server itself. This also helped me understand the practical value of the principle of least privilege: permissions should be limited to only what a resource actually needs.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/2c82353b-a1ec-4f15-9207-35eb1815d62a.png" alt="IAM policy JSON showing the restricted S3 upload permission" style="display:block;margin:0 auto" />

<h2>Automating the backup process</h2>
<p>Once the storage and permissions were configured, I moved on to automating the application data and backup process.</p>
<p>I created a directory at /var/app/data and configured a systemd service called data-generator.service. The service continuously wrote timestamped entries to an application log file every 30 seconds. I then configured it to start automatically when the server booted and restart if it stopped unexpectedly. This gave me a continuously changing set of application data to back up.</p>
<p>I then wrote a Bash script called backup.sh whose job was to compress the application data into a timestamped .tar.gz archive, upload that archive to the S3 bucket using the AWS CLI and record the result in /var/log/backup.log. After a successful upload, the script deleted the temporary archive from the server. I also included error handling so that a failed backup would be recorded in the log.</p>
<p>Finally, I used cron to schedule the script to run every ten minutes. This meant that the backup process could run automatically without requiring someone to execute it manually. Seeing the application log entries appear, the backup script run and the files arrive in S3 was one of the more satisfying parts of the implementation. It was the point where several individual configurations started working together as an actual system.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/6289ba1b-204f-4033-9ce0-6c8bf3fe0d59.png" alt="Terminal screenshot showing the running systemd service and application log entries" style="display:block;margin:0 auto" />

<img src="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/0d8175ae-121a-4977-9fde-f09f8120afc9.png" alt="Terminal screenshots showing the crontab entry" style="display:block;margin:0 auto" />

<img src="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/72bccf78-32e9-421e-a982-6415367f28b8.png" alt="successful backup logs" style="display:block;margin:0 auto" />

<h2>Creating an automated audit trail with Lambda and DynamoDB</h2>
<p>The next requirement was to make sure that every new backup was recorded automatically.</p>
<p>I created a DynamoDB table called backup-log, which would store information about each uploaded backup. I then configured an S3 event notification to respond whenever a new object was uploaded into the backups/ folder. The notification triggered an AWS Lambda function, which extracted details from the S3 event, including the backup filename, file size, upload timestamp and bucket name. It then wrote these details into DynamoDB as a new record.</p>
<p>What I found interesting about this part was that the services didn't need to communicate through a process that I manually initiated. S3 could generate an event, Lambda could respond to it, and DynamoDB could store the resulting information. This introduced me to the idea of event-driven architecture in a practical way. Each service had a specific responsibility, but together they formed an automated audit process.</p>
<p><strong>[IMAGE 4: Lambda function configuration showing the S3 trigger, followed by the DynamoDB table containing the backup records.]</strong></p>
<img src="https://cdn.hashnode.com/uploads/covers/6b00d2313244e4eeb7635e1b/4ebd19f0-c5d7-4880-81d6-7ffd6bcbb6cf.png" alt="" style="display:block;margin:0 auto" />

<h2>Testing the system and a lesson I didn't expect</h2>
<p>Getting everything configured was one thing. Making sure it behaved as required was another. One of the most important tests involved removing the NAT Gateway. After deleting it, I attempted to access an external website from the application server. The connection failed, as expected, because the server no longer had general internet access.</p>
<p>I then ran the backup script again. The upload to S3 still worked through the S3 Gateway VPC Endpoint. I also tested the IAM permissions by attempting to delete a backup object from the application server. The request was denied, which confirmed that the instance role did not have the required deletion permission.</p>
<p>There was another lesson I learned during the final stages of the project, and it had nothing to do with configuring an AWS service.</p>
<p>After completing the implementation, I started cleaning up the resources. Unfortunately, I deleted them before capturing all the required screenshots. I realised afterwards that I had missed the evidence showing the IAM policy and its attachment to the EC2 instance.</p>
<p>Rather than pretending the evidence was available, I made my research and possible recovery and found myself going to CloudTrail to investigate the original configuration. I found an "AssociateIamInstanceProfile" event containing the original instance ID and IAM instance profile information. I then recreated the relevant IAM policy and role, along with a temporary EC2 instance, so I could capture the configuration again.</p>
<blockquote>
<p>It was a small but useful lesson. I had spent time making sure the infrastructure worked, but I hadn't given the documentation and evidence collection the same attention. In a real engineering environment, being able to demonstrate what was configured, how it was tested and what happened during deployment is also part of the work.</p>
</blockquote>
<h2>What I would improve</h2>
<p>There are several things I would consider if I were extending this project beyond the original requirements.</p>
<p>For example, I would enable S3 bucket versioning to provide additional protection against accidental changes or deletion. I would also configure lifecycle rules to manage older backups and control storage costs over time.</p>
<p>I would introduce more detailed monitoring and alerts for failed backup attempts so that problems could be identified quickly. I would also review the encryption configuration and consider additional monitoring around access to the stored backup files.</p>
<p>These improvements would make the system more suitable for a production environment, where reliability, cost management and security need to be considered continuously.</p>
<h2>Conclusion</h2>
<p>This project gave me an opportunity to work with several AWS services in a connected environment rather than learning about them individually.</p>
<p>I worked with VPCs and subnets, EC2 instances, IAM roles, S3, Lambda and DynamoDB. More importantly, I got to see how networking, permissions and automation influence one another when building a cloud-based solution.</p>
<p>There were moments when I had to revisit configurations, troubleshoot problems and rethink parts of the implementation. Even the mistake of deleting resources before capturing all the evidence became a learning experience.</p>
<p>I'm still developing my understanding of cloud engineering, but projects like this make the concepts much easier to connect to real-world applications. It has given me a better appreciation of what goes into building infrastructure that is not only functional, but also secure, automated and maintainable.</p>
]]></content:encoded></item></channel></rss>