Friday, December 30, 2016

AWS S3 Data transfer - Kill the Colo ;-)

Transferring data from your premises or Colo infrastructure to AWS Cloud is not longer as difficult as it used to be ;-)

Besides dedicated links and physical transport of files, Amazon provides a pure internet solution (S3 Transfer Acceleration) to transfer files to Amazon Simple Storage Solution (S3) which might be enough for your needs. I will describe here my experience using this method from a manual perspective (no scripting this time) which should be enough for cases when for example you are moving to the cloud those big on premises or on Colo files.

Start by performing a test to see how faster this method will be in comparison with the direct upload method http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html.

I had to transfer 1 TB of data to S3 from Windows 2008 servers. Here is how I did it.

To transfer your files with S3 Accelerated Transfer Upload Speed select your bucket from AWS console | Properties | Transfer Acceleration | Enable | Get the accelerated endpoint which will work just as the regular endpoint; for example mys3bucketname.s3-accelerate.amazonaws.com.

You can use AWS CLI in windows just as well as in *nix systems ( http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-msi-on-windows ) to upload or download files.

See below for an example that shows how I configure AWS CLI, list what I have in S3, set the AWS CLI to use the accelerated endpoint and finally copy the data to the bucket.
C:\> aws configure
AWS Access Key ID [****************HTAQ]:
AWS Secret Access Key [****************J+aE]:
Default region name [None]: us-east-1
Default output format [None]: json

C:\> aws s3 ls
2016-12-24 06:25:01 mys3bucket1
2016-12-03 15:15:37 mys3bucket2

C:\> aws configure set default.s3.use_accelerate_endpoint true

C:\> aws s3 sync E:\BigData\ s3://mys3bucket2
I got 12MB/sec versus 1.2MB/sec at regular speed. I was able to transfer 1 TB in around 16 hours. The good thing is that the command behaves like rsync meaning that new files or addition to existing files will be the only data you will be transferring after that first attempt. This is good news when you are planning to move the infrastructure to the cloud as it minimizes the possible business disruption timeframe.

C:\> aws configure set default.s3.use_accelerate_endpoint true

C:\> aws s3 sync s3://mys3bucket2 D:\SmallToBeBigDATA\
WARNING: S3 transfer Acceleration has an associated cost. All I can say is: It is worth it. It costed us $8.28 to transfer above 1TB of data from an on premises Windows Server to an EC2 hosted Windows Server via S3.

AWS Inventory - Audit the Cloud Infrastructure

Update 20180103: I just created a new PR that adds support for IAM, namely it lists users and the policies assigned to them. In order to use this version you need to add a new policy to the role used to run the Lambda:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:ListUsers",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:ListUserPolicies",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:ListAttachedUserPolicies",
            "Resource": "*"
        }
    ]
}
Update 20171217: I just created a new PR with a couple of fixes:
* Lambda functions must finalize within a short period of time and therefore the amount of snapshots should be an external environment variable.
* Reserved IP addresses might not be in use in which case we should show the instance id as empty, otherwise we get an exception.
How difficult is to audit your AWS Cloud Infrastructure?

Instances, volumes, snapshots, RDS, security groups, elastic IPs and beyond. A single report to get all the invaluable information that will keep you informed to make critical and quick decisions.

The guys from powerupcloud shared an initial script in their blog which they put in github. I forked it and after some tweaks found it so useful that I decided to ask the author for a pull request. The new Lambda Function:
* Adds support for environment variables
* Adds security groups listing
* Removes hardcoded and non generic names
* Corrects some comments
* Retrieves the ownerId instead of hardcoding it
* Adds the description for volumes for clearer identification
* Lists the Elastic IPs with the instanceId the are assigned to for clearer identification
* Has a TODO ;-: Naming conventions should be established
Here is a quick start:
  1. Create IAM role | Name:Inventory; Role Type: AWSLambda; Policy: EC2ReadOnly, AmazonS3FullAccess, AmazonRDSReadOnlyAccess
  2. Create S3 bucket | Name: YourInventoryS3Name
  3. Create Lambda | Name: YourInventoryLambda; Description: Extracts the AWS Inventory; Runtime: Python; Role: Inventory; Timeout: 3 min; environment variables: SES_SMTP_USER, SES_SMTP_PASSWORD, S3_INVENTORY_BUCKET, MAIL_FROM, MAIL_TO
  4. Schedule the Lambda: Select Lambda | Trigger | Add Trigger | CloudWatch Events - Schedule | Rule Name: InventorySchedulerRule; Rule Description: Inventory Scheduler Rule; Schedule Expression: cron(0 14 ? * FRI *) if you want it to run every Friday at 9AM ET| Enable Trigger | Submit
To be able to send emails from AWS you need to allow Amazon Simple Email Service (SES) to send emails for some emails in your domain:
  1. Verify the domain: SES | Manage Entities | Verify a New Domain ; yourdomain.com
  2. Follow the instructions. To complete the verification you must add several TXT/CNAME records to your domain's DNS settings
  3. Verify email addresses: SES | hit the domain | Verify a New Email Address | Confirm from email
  4. After getting confirmation go to SES | SMTP Settings | Create my SMTP Credentials | Provide a username like ‘anyusername’ for example and Amazon will show you the credentials (SMTP Username and SMTP Password). Keep them in safe place. You can also download them. This is the last time AWS will share them with you.
You can clone/fetch/checkout my latest version from my fork: Really useful stuff. Thanks PowerUpCloud for the quick start!

Tuesday, December 27, 2016

AWS EC2 DR - cross region replication for COB via Lambda Function

It happens. The continuity of the business (COB) will be compromised if there is not a sound Disaster Recovery (DR) plan. DR is the ultimate goal of a business continuity plan (BCP). To accomplish this task for Amazon hosted machines we could use a CloudFormation stack template as we did for the in region backup. However, if we just need to replicate all snapshots created in a region (in region backup) to another, then all we need to build is a Lambda function and schedule it.

Create an AWS Identity and Access Management (IAM) Role: AWS Console | Services | IAM | Roles | Create New Role | Name: CrossRegionReplication | Next | Select Role Type: AWS Lambda | Attach Policy: AmazonEC2FullAccess | Create Role

Create a Lambda Function: AWS Console | Services | Lambda | Create a Lambda Function | Configure Function | Name: replicateAll; Description: Cross Region Replication; Runtime: Python; Paste the code from aws-cross-region-replicate-all-lambda.py script and customize it to your needs; Role: CrossRegionReplication; Timeout: 5 min; | Next | Create Function Test the Lambda: You can test the whole function or just part of it using the “Test” button. Very useful for example to see what it will do if you comment the copy_snapshot() statement.

Schedule the Lambda: Select Lambda | Trigger | Add Trigger | CloudWatch Events - Schedule | Rule Name: ReplicationSchedulerRule; Rule Description: Replication Scheduler Rule; Schedule Expression: rate(1 hour) | Enable Trigger | Submit

What did we do? In the last two posts I have shown how to create a bare minimum BCP/DR for AWS EC2. There is a lot more to have ready in order to make sure that a DR datacenter can operate as the new live datacenter, however having the data volumes available is the absolute first step. Out of the snapshots we could at least manually rebuild the environment. Culture first.

CloudFormation AWS EC2 BCP - in region backups for COB

To achieve continuity of business (COB) you need a business continuity plan (BCP). A crucial part of such plan is to have available backups.

Amazon Web Services (AWS) offer Amazon Elastic Compute Cloud (EC2) instances that can be crash-consistent backed up into Amazon Simple Storage Solution (S3) via Amazon Elastic Block Store (EBS) snapshots.

Here I show how to use the out of the box provided AWS CloudFormation Template for EBS Snapshot Scheduler. No need to deal with AWS CLI, lambda functions, task scheduling, extra dedicated instances, IAM policies and permissions.

From https://s3.amazonaws.com/solutions-reference/ebs-snapshot-scheduler/latest/ebs-snapshot-scheduler.pdf locate the “View Template” button which will download the ebs-snapshot-scheduler.template file (there is also a “download the template”). Rename your template and customize it. Below is an example of relevant fields you might want to change up front:
...
    "Description": "EBS Snapshot Scheduler for Test Environment,
...
        "DefaultRetentionDays": {
            "Description": "Default Retention Period (number of days)",
            "Type": "String",
            "Default": "7"
        },
...
        "CustomTagName": {
            "Description": "Custom Tag Name for testing environment backup",
            "Type": "String",
            "Default": "scheduler:ebs-snapshot:test"
        },
...
        "DefaultTimeZone": {
            "Type": "String",
            "Default": "US/Eastern",
...
        "AutoSnapshotDeletion": {
            "Description": "Enable auto-delete EBS snapshots after retention period.",
            "Type": "String",
            "Default": "Yes",
...
        "HistoryDDBTableName": {
            "Description": "History DynamoDB Table Name for Testing Environment",
            "Type": "String",
            "Default": "Scheduler-EBS-Snapshot-History-Test"
        },
        "PolicyDDBTableName": {
            "Description": "Policy DynamoDB Table Name for Testing Environment",
            "Type": "String",
            "Default": "Scheduler-EBS-Snapshot-Policy-Test"
        },
...
The Launch Solution button/link will allow you to define a new CloudFormation Stack (the collection of resources to be provisioned which are defined in a template file). Alternatively use Console | Services | CloudFormation | Create Stack | Choose a Template. Make sure to check what zone you are in the console UI and as usual change it to the correct one.

Select “Upload a template to Amazon S3” and upload the one you created above.

Type a stack name like EBSSnapshotSchedulerTest. If you don’t pick a name the default would be EBSSnapshotScheduler. You can configure from the UI most of the previously mentioned configuration but you might want to do that from the template anyway to keep your configuration files current and correctly versioned.

Press Next all the way through to go with defaults, check “I acknowledge that AWS CloudFormation might create IAM resources.” and hit Create.

Wait until the status of your stack is CREATE_COMPLETE and tag the instances you want to include in the backup, for example the below tag will use defaults:
scheduler:ebs-snapshot:test=true
The below tag will mean take two daily snapshots, one at midnight and one at noon in UTC zone and keep them for 7 years (2555 days):
scheduler:ebs-snapshot:prod:00 0000;2555;utc;all
scheduler:ebs-snapshot:prod:12 1200;2555;utc;all
You can deactivate by removing the tags.

To get an idea about how much time you saved take a look at your template from the AWS CloudFormation Designer:



That’s probably 24 hours worth of work and troubleshooting. It is worth to be said that I published a scheduled Lambda that will take care of cross region replication, which demonstrates that we could achieve the same most important goal for in-region backup in a simpler way. Of course you will need to code a bit for that solution. Hopefully the creators of the CloudFormation EBS Snapshot Scheduler will either add replication as a feature or will build a new CloudFormation to take care of replication for the masses.

Adding new backup policies should be straightforward. You just need to focus on customizing a new template and following the steps described above. For example, for a typical production 7 years retention backup we can use the below selecting as Stack name EBSSnapshotSchedulerProd. Don’t forget to tag the instances you want to include in the backup “scheduler:ebs-snapshot:prod=true”. To deactivate it use “scheduler:ebs-snapshot:prod=none” or just remove it:
...
    "Description": "EBS Snapshot Scheduler for Production Environment,
...
        "DefaultRetentionDays": {
            "Description": "Default Retention Period (number of days)",
            "Type": "String",
            "Default": "2555"
        },
...
        "CustomTagName": {
            "Description": "Custom Tag Name for production environment backup",
            "Type": "String",
            "Default": "scheduler:ebs-snapshot:prod"
        },
...
        "DefaultTimeZone": {
            "Type": "String",
            "Default": "US/Eastern",
...
        "AutoSnapshotDeletion": {
            "Description": "Enable auto-delete EBS snapshots after retention period.",
            "Type": "String",
            "Default": "Yes",
...
        "HistoryDDBTableName": {
            "Description": "History DynamoDB Table Name for Production Environment",
            "Type": "String",
            "Default": "Scheduler-EBS-Snapshot-History-Prod"
        },
        "PolicyDDBTableName": {
            "Description": "Policy DynamoDB Table Name for Production Environment",
            "Type": "String",
            "Default": "Scheduler-EBS-Snapshot-Policy-Prod"
        },
...
The ebs-snapshot-scheduler is open source.

If you need to take multiple backups within a day you will need to retag your instance but you can still use just one CloudFormation stack. I have opened a feature request to avoid adding an extra tag per snapshot creation time leveraging cron expressions.

Thursday, December 22, 2016

Working with Microsoft SQL Server (MSSQL) from Ubuntu Linux Terminal

Change your TDS version below as needed. You can add the $password directly at the end if you know what you are doing.

Wednesday, December 21, 2016

NetSuite tokenPassport Generator

If you need to test the NetSuite SOAP API (SuiteCloud) with tools like SOAPUI you might need a quick way to generate the tokenPassport node:

Friday, December 09, 2016

The lean http to https redirection in IIS

The leanest way to run a secure website from IIS is to serve just one application. If you can live with that, then redirecting HTTP to HTTPS traffic should not be as painful:
  1. Add IIS Redirect Module: Server Manager | Roles Summary | Add Roles | Web Server - Common HTTP Features - HTTP Redirection
  2. Redirect http to https: IIS Manager | Select server node | Bindings | Remove port 80 binding | Default Web Site node | Bindings | Set just port 80 binding | HTTP Redirect | Redirect to a specific https:// destination | Status code =Found (302)
I prefer to use the "Found" (302 option) rather than "Permanent" (301 option) just in case I want to change my domains in the future.

Simplicity rules.

Followers