Resizing EBS Volumes for Your EC2 Instances: A Step-by-Step Guide

Running out of space on your Amazon EC2 instance? Don't worry, you're not alone. Thankfully, with Elastic Block Store (EBS) volumes, expanding your storage capacity is a straightforward process. In this guide, we'll walk you through the steps to seamlessly resize your EBS volumes and ensure your EC2 instance has ample room to grow.

Why Resize EBS Volumes?

EBS volumes provide persistent block storage for your EC2 instances. As your applications and data grow, you might find the initial storage allocation becoming insufficient. Resizing EBS volumes allows you to increase the storage capacity without the need to create a new instance or migrate data, minimizing downtime and disruption.

Steps to Resize Your EBS Volume:

1. Stop Your Instance: Navigate to the EC2 Instances console within the AWS Management Console and stop the instance whose EBS volume you want to resize. Note the availability zone of your instance – this is crucial for later steps. Also, make a note of the mount point of the volume (e.g., /dev/sdxx).

2. Create a Snapshot: Go to the EBS Volumes console and locate the volume attached to your stopped instance. Select the volume and choose the "Take Snapshot" option. This creates a point-in-time backup of your data.

3. Create a New Volume from the Snapshot: Find the newly created snapshot in the EBS Snapshots console. Select it and click "Create Volume." Specify the desired increased size for the new volume and ensure you select the same availability zone as your EC2 instance.

4. Detach and Attach Volumes:

   • Head back to the EBS Volumes console.
   • Select the old volume, choose "Actions," and then "Detach Volume."
   • Select the new volume, choose "Actions," and then "Attach Volume."
   • Choose your instance from the list.
   • In the "Device" field, ensure you enter the correct mount point you noted in step 1 (e.g., /dev/sdxx).

5. Start Your Instance: Restart your EC2 instance from the EC2 Instances console.

6. Extend the Filesystem:

   • Once the instance is running, SSH into it.
   • Run df -h to list partitions. You'll see the new volume, likely mounted at /dev/xvda1 (or similar). Note that the displayed size won't reflect the increased capacity yet.
   • Extend the filesystem to utilize the full volume size by running:
     Bash

     resize2fs /dev/xvda1 
     (Replace /dev/xvda1 if your volume has a different mount point.)

Important Tips:

• Snapshots Are Your Friends: Always take a snapshot before resizing volumes, ensuring you have a rollback point in case of unexpected issues.
• Choose the Right Volume Type: If your workload demands high performance, consider using Provisioned IOPS SSD (io1) or General Purpose SSD (gp3) volumes for optimal results.
• Monitor Storage Usage: Regularly monitor your EBS volume usage to ensure you have enough headroom and plan for future resizing.

By following these steps, you can effortlessly resize your EBS volumes and scale your EC2 instances to meet the demands of your growing applications and workloads.

Mastering Puppet: A Guide to Configuring the Puppet Master and Client

Puppet is a powerful configuration management tool that automates the process of managing your infrastructure. Setting up a Puppet Master and its clients can seem daunting, but with this guide, you'll be equipped to handle the initial configuration with ease. This blog will walk you through the steps needed to set up a Puppet Master and client, ensuring a smooth and secure connection between them.

Step 1: Initial Setup for Both Master and Client

Downloading and Installing Needed RPM

Before you begin, ensure that both the Master and the client machines have the necessary RPM installed. This can be done by running:

rpm -ivUh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

This command will install the EPEL (Extra Packages for Enterprise Linux) repository, providing additional packages for your setup.

Step 2: Installing the Puppet Server and Client

Master: Installing Puppet Server

On your Master machine, install the Puppet Server with Yum:

yum install puppet-server

Client: Installing Puppet

On the client machine, install the Puppet client software:

yum install puppet

Step 3: Configuring Hostnames and Network

Ensure that the Master and client can communicate with each other by setting up the hostnames correctly.

Edit the Hosts File

Add the following entries to the /etc/hosts file on both the Master and client:

xxx.xxx.xxx.xxx master.puppet.com

xxx.xxx.xxx.xxx client.puppet.com

Replace xxx.xxx.xxx.xxx with the appropriate IP addresses.

Test the Connection

Test the connectivity between the Master and client using the ping command:

ping -c 3 client.puppet.com

ping -c 3 master.puppet.com

Step 4: Setting Up Iptables

For secure communication, you need to ensure that the correct port is open on both the Master and client.

Modify Iptables Rules

You can either disable Iptables or open port 8140, which Puppet uses for communication:

iptables -A INPUT -p tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT

Step 5: Starting the Puppet Master Server

With the configurations set, it's time to start the Puppet Master.

Start the Server

On the Master machine, start the Puppet Master service:

/etc/init.d/puppetmaster restart

Step 6: Client Certificate Signing

Puppet uses a certificate-based authentication system. The client will request a certificate from the Master, which needs to be signed.

Check for Signed Certificates

From the client machine, initiate a certificate signing request:

puppetd --server=master.puppet.com --waitforcert 60 --test

Sign the Client's Certificate

On the Master, list all unsigned certificates:

puppetca --list

Sign the client's certificate:

puppetca --sign client.puppet.com

Step 7: Creating Configuration for Clients

With the infrastructure in place, you'll now need to define the desired state of your client systems in the Puppet Master.

Edit the Manifest File

Add configurations to /etc/puppet/manifests/site.pp on the Master. Here's a sample configuration:

# Create "/tmp/testfile" if it doesn't exist.

file { "/tmp/outside":

ensure => present,

mode => 644,

owner => root,

group => root

}

class test_class {

file { "/tmp/testfile":

ensure => present,

mode => 644,

owner => root,

group => root

}

}

package {

'httpd':

ensure => installed }

service {

'httpd':

ensure => true,

enable => true,

require => Package['httpd']

}

# tell puppet on which client to run the class

node client {

include test_class

}

Conclusion

Congratulations! If you've followed these steps without error, your Puppet Master and client are now configured and communicating securely. With your infrastructure now under Puppet's management, you're set to automate your configurations, ensuring consistency and reliability across your environment. Remember, Puppet is incredibly powerful and flexible. Continue exploring its capabilities to fully harness its potential in managing your infrastructure.

Building a Custom NAT Server on AWS: A Step-by-Step Guide

Network Address Translation (NAT) servers are essential components in a cloud infrastructure, allowing instances in a private subnet to connect to the internet or other AWS services while preventing the internet from initiating a connection with those instances. This blog provides a detailed guide on setting up a NAT server from scratch in an AWS cloud environment.

Step 1: Launching an AWS Instance

Start a t1.micro instance:

• Navigate to the AWS Management Console.
• Select the EC2 service and choose to launch a t1.micro instance.
• Pick an Amazon Machine Image (AMI) that suits your needs (commonly Amazon Linux or Ubuntu).
• Configure instance details ensuring it's in the same VPC as your private subnet but in a public subnet.

Step 2: Configuring the Instance

Disable "Change Source / Dest Check":

• Right-click on the instance from the EC2 dashboard.
• Navigate to "Networking" and select "Change Source / Dest Check."
• Disable this setting to allow the instance to route traffic not specifically destined for itself.

Security Group Settings:

• Ensure the Security Group associated with your NAT instance allows the necessary traffic.
• Typically, it should allow inbound traffic on ports 80 (HTTP) and 443 (HTTPS) for updates and patches.

Step 3: Configuring the NAT Server

Access your instance via SSH and perform the following configurations:

Enable IP Forwarding:

1. Edit the /etc/sysctl.conf file to enable IP forwarding. This setting allows the instance to forward traffic from the private subnet to the internet.

   sed -i "s/net.ipv4.ip_forward.*/net.ipv4.ip_forward = 1/g" /etc/sysctl.conf
   

2. Activate the change immediately:

   echo 1 > /proc/sys/net/ipv4/ip_forward
   

3. Confirm the change:

   cat /etc/sysctl.conf | grep net.ipv4.ip_forward
   

   Expected output: net.ipv4.ip_forward = 1

Configure iptables:

1. Set up NAT using iptables to masquerade outbound traffic, making it appear as if it originates from the NAT server:

   iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
   

   This command routes all connections reaching eth0 (the primary network interface) to all available paths.

2. Allow traffic on ports 80 and 443 for updates and external access:

   iptables -A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
   iptables -A INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT
   iptables -A FORWARD -i eth0 -j ACCEPT
   

Step 4: Routing Configuration

Configure Route Tables:

• In the AWS Console, go to the VPC Dashboard and select Route Tables.
• Modify the route table associated with your private subnet:
  • Add a route where the destination is 0.0.0.0/0 (representing all traffic), and the target is the instance ID of your NAT server.
• Modify the route table associated with your NAT instance:
  • Ensure there's a route where the destination is 0.0.0.0/0, and the target is the internet gateway of your VPC.

Conclusion

With these steps, you've successfully created a NAT server in your AWS environment, allowing instances in a private subnet to securely access the internet for updates and communicate with other AWS services. This setup is crucial for maintaining a secure and efficient cloud infrastructure. Always monitor and maintain your NAT server to ensure it operates smoothly and securely. Currently there are managed NAT server services from AWs which we can use for production grade environments.

NextCloud Setup with Docker

One of the most commonly used self-hosted alternatives for cloud storages. Now it's easy to deploy with dockers. Following docker file and Nginx configuration can be used to deploy the nextcloud application behind the Nginx proxy server with SSL termination. 

we can bring up and bring down the containers with the following commands

docket-compose up -f
docker-compose down

===========

version: '2'
#volumes:
#  nextcloud: /root/nextcloud/ncdata
#  db: /root/nextcloud/mysql
services:
  db:
    image: mariadb:10.5
    restart: always
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    volumes:
      - /root/nextcloud/mysql:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=XXXXXXXXX
      - MYSQL_PASSWORD=XXXXXXXX
      - MYSQL_DATABASE=XXXXXXXX
      - MYSQL_USER=XXXXXXXX
  app:
    image: nextcloud
    restart: always
    links:
      - db
    volumes:
      - /root/nextcloud/ncdata:/var/www/html
    environment:
      - MYSQL_PASSWORD=XXXXXXXX
      - MYSQL_DATABASE=XXXXXXXX
      - MYSQL_USER=XXXXXXXX
      - MYSQL_HOST=XXXXXXXX
      - NEXTCLOUD_TRUSTED_DOMAINS=abc.xyz.aa
      - OVERWRITEHOST=abc.xyz.aa:XXXX
      - OVERWRITEPROTOCOL=https
        
  web:
       image: nginx
       restart: always
       ports:
         - 8082:8080
       links:
         - app
       volumes:
         - /root/nextcloud/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
         - /root/nextcloud/cert:/etc/cert
===========
Nginx Configuration file
===========

server {
  listen 80;
  server_name abc.xyz.aa;
  return 301 https://$server_name:8080$request_uri;
  add_header X-Content-Type-Options              "nosniff";
}
server {
  listen 8080 ssl;
  server_name abc.xyz.aa;
  ssl_certificate /etc/cert/abc.xyz.aa.crt;
  ssl_certificate_key /etc/cert/abc.xyz.aa.key;
  ssl_prefer_server_ciphers on;
  location / {
  proxy_pass http://app;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
 }
} 
===========

Increases swap in azure linux machine

In Azure to create a swap file in the directory that's defined by the ResourceDisk.MountPoint parameter, you can update the /etc/waagent.conf file by setting the following three parameters:

ResourceDisk.Format=y
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=xx

Note The xx placeholder represents the desired number of megabytes (MB) for the swap file.
Restart the WALinuxAgent service by running one of the following commands, depending on the system in question:

Ubuntu: service walinuxagent restart
Red Hat/Centos: service waagent restart

Run one of the following commands to show the new swap apace that's being used after the restart:

dmesg | grep swap
swapon -s
cat /proc/swaps
file /mnt/resource/swapfile
free| grep -i swap

If the swap file isn't created, you can restart the virtual machine by using one of the following commands:

shutdown -r now
init 6

Qubole : Load Multiple tables to Qubole Hive table from a Data Store

API call to Load Multiple tables from a Qubole Data Store to Hive table. 

[rahul@local qubole]$ cat /databasescript 

#!/bin/bash

#Qubole API Key

AUTH="***********"

#Database Name

DB_NAME="***********"

#Host Name

DB_HOST="***********"

#User Name

DB_USER="***********

#Password 

DB_PASS='***********'

echo $DB_PASS

## request table import from tap;

function tableImport() {

request_body=$(cat <<EOF

{

   "command_type":"DbImportCommand",

   "mode":"1",

   "hive_serde":"orc",

   "hive_table":"<HIVE TABLE NAME>.$1",

   "dbtap_id":"$2",

   "db_table":"$1",

   "db_parallelism":"1",

   "use_customer_cluster":"1",

   "customer_cluster_label":"Qubole_Data_Import",

   "tags":[" Data"]

}

EOF

)

echo $request_body

   curl -X POST \

-H "X-AUTH-TOKEN: $AUTH" \

-H "Content-Type:application/json" \

-d "$request_body" https://api.qubole.com/api/v1.2/commands/

}

##register database with tap

request_body=$(cat <<EOF

{

  "db_name":"$DB_NAME",

  "db_host":"$DB_HOST",

  "db_user":"$DB_USER",

  "db_passwd":"$DB_PASS",

  "db_type":"sqlserver",

  "db_location":"on-premise",

  "gateway_ip": "***********",

  "gateway_port": "***********",

  "gateway_username": "***********",

  "gateway_private_key": "***********"}

EOF

)

echo $KEY

ID=$(curl -s -X POST \

-H "X-AUTH-TOKEN: $AUTH" \

-H "Content-Type:application/json" \

-d "$request_body" https://api.qubole.com/api/v1.2/db_taps/ | jq .id)

#get the tables and call import

curl -s -H "X-AUTH-TOKEN: $AUTH" \

     -H "Content-Type:application/json" \

     https://api.qubole.com/api/v1.2/db_taps/$ID/tables | jq -r .[] | while read x; do  tableImport $x $ID; done

# can't delete the tap at the end unless we continuously poll for no active jobs;

STATUS="null"

while [ "$STATUS" = "null" ]

do

STATUS=$(curl  -s -X DELETE \

 -H "X-AUTH-TOKEN: $AUTH" \

 -H "Content-Type:application/json" \

 https://api.qubole.com/api/v1.2/db_taps/$ID | jq .status)

echo -n "."

sleep 5

done