Getting to manage AWS with Python - The power of Infrastructure as Code

 Hello all,

    In today´s article we are going to start working  with two of the most exciting and powerful platform I ever seen: Python and AWS.
  Both of it needs no introduction, so let´s getting started:

In Python, we are going to install the AWS interface called Boto. I have no idea from where this name came from, I´ll be looking for it later, but, Boto, the way I know, is an animal found in Brazil, in Amazon forest specifically and is known as a magic animal. Well, the animal itself I know and it is real, it is some kind of dolphin´s cousin, very similar. About the magic, is up to you!
 To install it on Python just hit pip install boto:

The next step is getting safety. When you created an account in Amazon console, it is allowed for you to save your credentials in a csv file. If you didn´t yet, we can do it now:

  In the AWS console dashboard, go to your account and "My Security Credentials":

   You can create a new Access Key.

   In my case, I have already done this,  I can create it again if necessary. Luckily, I need not.

  Once created your Access Key, you can download the file in CSV format. Keep it safe! Now we are going to use it in our Python code.
 First of all, create a file in /etc called boto.cfg
nano /etc/boto.cfg

Inside the file, put your Access Key, like this:

Save the file.

 Now we are going to explore some AWS Python commands:

Open up a new file and create a script like this on:

This very simple script is able to accomplish something amazing. Let´s explain line by line.

First we need to import boto. 
Then, we create a variable we call s3 and give it the value "boto.connect_s3(). This is actually a function that allows us to connect to the AWS console using Python.
 Next line we created another variable called bucket. Then, we called the variable defined above, s3, and use the command create_bucket. We also passed as argument the bucket´ss name. As a comment, we are saying that a bucket must have an unique name.
 Now that we have created our bucket, we are going to put something inside our bucket. We are going to store something there. The way we do that is:
 key = bucket.new_key("Directory/file")

We just created one more variable above and gave it as value the directory and file we would like to store in the Cloud. This can be anything.
 Now, we are going to use this variable in conjunction with the command set_contests and we will sent to the Cloud some file.
key.sent_contents_from_filename("Your PC Directory/ Your file")

Lastly, we are going to set permission.

key.set_acl('public-read')

Now, we are going to execute our script and see what is might accomplish in AWS console!

And here we go!  We have just created a brand new bucket from command line. Let´s see what´s inside this bucket:

That´s right!! We just upload our own AWS script to a bucket in the Cloud. We can see highlighted in yellow that, as we asked to, the script created a root directory and a DevOps sub directory. Inside we can see our file.

 That´s it guys. This is a small step but the power behind it is unmeasurable and our goal here is to explore as much as possible all the possibilities here.

Cheers!

How to create a VPC on Amazon AWS - Part 02 - "Freeing my Private Server"

In the first part of this tutorial we learned how to create a brand new VPC on Amazon AWS. On our environment we had two servers: a Web server which we allow internet connections and a DB server which you choose keep closed.
  Well, it is a comon setup. But, sometimes we need our server to access the Internet. I am not telling you that your private server will going to have access from the Internet, otherwise, it wouldn´t be private anymore. But, what if I need this server to access a repository on the Internet do perform updates instead transfering this large amount of data manually. Would be much more practical wouldn´t be?
 In this tutorial we are going to see how to allows our server to access the Internet and still keeping hide from the Internet point of view.
  Amazaon allows us to create a NAT instance which is an EC2 instance able to perform NAT.

This time I´m going to teach you how to create the instance as this is a different instance and not a common one:

Go to the ES2 Dashboard, go to Launch Instance then go to Community AMIs and on the search bar above look for NAT. The first instance the appear bellow is the best option.

In the Step 2, amanzon gives you the chance to change your Instance type. We are going to keep unchanged because you want this for free.

Step 3 we need to change the Subnet to the Public subnet, in case the private was selected:

In step 4 you don't change anything, just hit next.
Step 5 you give it a name.
Step 6 is time to setup a security group. It depends on you. You can create a brand new Security Group or you can use one you´ve created before.
Step 7 you Launch the instance. Check you Key and done!

If you have never create an EC2 instance before, this might be a little difficult. Please, take a look on how to create EC2 instance first.

And here we have our NAT instance up and running. One last change we need to do is on Network:

In Change Source/Dest Check, we are going to disable it. Go ahead and do it.

 Also , we need to setup a route.  This time, we don´t need actually to create a new route table, it enough to modify our default routing table like we are going to do bellow:

Go to your VPC dashboard, find Route Tables on the left. Select your default route and under Routes add a new entry. The destination is the whole Internet 0.0.0.0/0 and the Target will be our NAT Instance.
   That´s it!

Considering you have you public server up and running and you are able to connect to it, you will be able to connect to your private server as well. After all, they are on the same 10.0.0.0/16 network.
  The difference from our first Lab from this one is that now our Private server is able to access the Internet.

As we can see from the image above, we are connected through SSH to our private server 10.0.2.251 and we are able to ping the Internet IP address 200.221.2.45. This validate our Lab in terms of NAT Instance.

How to create a VPC on Amazon AWS

 VPC stand for Virtual Private Cloud and it is a Data Center you can create on the Cloud using Amazon AWS.
  There are some required steps you should know in order for to have a up and running environment on the Cloud and that´s our goal here.
  First of all you need to have, of course, a valid user on the Amazon console which you can access here. If you don´t have an account yet, I strongly encourage you to create. Of course, considering you have interest in AWS, as I am.
  Another very important thing to learn when it comes to AWS is what you can do without paying anything. In the past, I have created an account and start digging through Amazon services without orientation and at the and of my free period, I received a fee. It was not a absurd, but I did not intend to pay anything, I was just trying it. At that time, I talked to an Amazon representative and I did pay anything by the way. Therefore, I recommend you to pay attention on this because it might change sometime and you may end up with a Bill in your hand and have to pay it.
  The nice thing about AWS is that you can actually use it without paying during your free eligible period, or at least, paying only a few dollars.
 Considering you have successfully logged in the Amazon Console, if this is your first time, you are going to see this:

 Ow yeah!! Each icon is an Amazon service and yes, Amazon has been one the most innovative company.
 I felt in love for this company about 3 years ago when I was called for an interview. I didn´t  know the company too much and I got really surprised. Sadly, I didn´t make it on the interview, but I did learn a lot about Cloud since then.
 Back to our objective here, once you get on this amazing portal, you need to scroll down and find, under Networking, the VPC icon. By clicking on it, you will be presented to the VPC dashboard with a lot of information. To be more effective here, I will focus on the things we are going to use only.
 You have a "Start VPC Wizard" which I never used, and I recommend you to do the same. The idea here is learn in detail and Wizards tends to let things easier.
 Under the "Start VPC Wizard" you have some web links and the first  of then is "VPCs". You may have a number 1 on it, this indicate you already have one VPC created. In my case, I have zero because I deleted my default VPC and I have the  felling I need to call Amazon if I want it back. As I don´t, I will not call.
 By click in "VPCs" whether you have 1 or 0, you can create a brand new VPC from scratch and that´s the idea here.By the waay, you can create up to 5 VPC per Region.
 When you click in "Create a VPC, you´ll receive a window where you can give your VPC a name and a CIDR block. If you dont know what CIDR block is I recommend you to read some paper about it. It is very important and don´t make any sense to know something about VPC if you don´t know it.
 In short, CIDR is the IP addressing your are going to use inside your VPC.

 I´ll call it MyVPC and I´ll allocate the 10.0.0.0/16 CIDR according to the RFC 1918. You can´t  use larger mask here then that and I don´t recommend you use a smaller one.
 In tenancy, let it as Default, the other option will cost you a lot of money.
 If the brand new VPC doesn´t appear right after you have created, don´t worry. Refresh the page and wait a few seconds.
 As soon as it has been created, it will appear as available.
 A few important information about VPC is Subnets, Route Tables, Network ACLs, Security Group and Internet Gateway.
 Some of this will be automatic created for you like Routing Tables, Security Groups and Network ACLs but we are going to created our own version of each of this.
  The first thing I believe make more sense to create and, as you can see, the system does not create a default for you, is Subnets.
 Subnets is what gives VPC a reason and it is where we a going to put our hosts. To create one or many, go to the Subnets left side menu and click in "Create Subnet".

Name tag is a name you can give.This can be anything. I like the example I saw on the Internet where the name contain the CIDR block to this subnet, the availability Zone and an indication of what this Subnet is about. In my case, I dicided to use 10.0.1.0/24 as Public. Which means, the servers I will put on this subnet will have access from the Internet.
 You may be asking: But this kind of IP address is not routable on the Internet and you are right about that. We´ll see how to solve it. By now, lets determine this Subnet as our Public subnet or DMZ.
  The process may be finished here, depending on what you want to do. In my case, I´ll create another Subnet and I´ll call it Private.  I want to create an environment where some servers will be reached from the Internet and others will not.
 So, to create the Private network, repeat the process we just made. Take care to use a different Subnet block. In my case, I´ll use 10.0.2.0/24.

We can see that, I used another  IP address block, Called it Private and put it in another Availability Zone. Which is a very good idea. This gives my environment, redundancy.
 An importan concept about AWS is each Availability Zone means one Subnet, you can´t have more then one subnet per AZ.

  As I mentioned before, one of my Subnet will be Public. To do so, I need to change on configuration parameter.

Go to Subnet Actions and in "Modify auto-assign IP setting" just click in "Enable auto-assign public IP4 address".

By doing so, when you designate a host to this Subnet, it will be given to this host an valid public IP address and it will be accessible from anywhere.
 Well, so far we have a VPC created, our VPC has two subnet one we called Private and another one we called Public. Our public subnet has a feature that allows it to get public IP address. Our environment is almost done!

Another very important concept in Networking is Routes. Routes allows our hosts to send packets back and forth depending on the origin or destination.
 As you can see, when we created our VPC, a default route have been created automatically. Let´s take a look on it:

   As we can see from the image above, our route table has only one route. This make sense, after all we have only the Network 10.0.0.0/16 on the environment. This means that our hosts will be able to reply IP packets to this network and that´s it. If a packet comes from a different network, the hosts will discard them. After all, they don't know  where to send it.
 To solve this problem first of all we are going to create a Gateway. Gateway is an entity that allows hosts communicate between networks. When a hosts need to send a packet to another host and this hosts belong to another network, this packet needs to be delivered using a Gateway.

To create a Gateway on this environment, just follow the image bellow:

Bu hitting "Create Internet Gateway" you can give your Gateway a name and then hit Yes, Create.

Therefore, when the Internet Gateway is created, it was Detached from your VPC. You need to Attach it by clicking "Attach to VPC".

If you have more then one VPC, make sure to select the correct one. If you are good to go, hit Yes, Attach.

This is the final result and  now we have a Internet Gateway to serve your VPC.

But we still need routes. That´s the next task. Let´s create a brand new Routing Table:

Give it a name and associate it to your VPC. You now have two Routing table. The default one and the one you just created.

Let´s edit the one we created.

Before editing, we can see that we only have the default route to our 10.0.0.0/16 Network. What we are going to do here is to create a new route. To do that, we need to hit the "Edit" button and "Add another route".

By hitting "Add another route", a new line appear. As we are going to create a route to the whole Internet, we need to use the 0.0.0.0/0 notation. This means Everything. By clicking on the next box, the Internet Gateway we have created appear as option. Make sure you are using the right one, in case you have more then one.

This will be what your routing table will look like. Now you have two routes. One the the local Subnet and one to the whole Internet.

 Well, we need to make it better. Our Routing table, in terms of internal network, is summarized in 10.0.0.0/16. Therefore, we have two Subnets behind it. The routing table needs to know which one to send.
 We can do this in the "Subnet Associations" and then "Edit" button:

By hitting "Save", our routing table will be able to decide to which Subnet it must reply.

We need one last step in order for to have our environment ready. We need to create a Security Group.

Again, we are going to ignore the default configuration AWS have made, and we are going to create a new Security Group.

Security Group is somehow a Firewall that protects our environment. It is important to say that Security Group only permit packets, you can´t deny anything explicitly. Actually, an implict Deny will exist at the end like a Firewall does. If you need to play with permit and deny interchangeable, ACL is the right thing for you.

 This first Security Group will be protecting our DMZ or Public network. We will be allowing only SSH,ping and HTTP connexions to our DMZ server. Anything else will be blocked:

By selecting the Security Group we just created, go to Inbound Rules and in Custom Rules and select the service you want to allow. You can add another rule by hitting "Add another rule" button.
 One very import thing here is the "Source" box. As you may not know the source, it is advisable to allow the whole Internet again: 0.0.0.0/0. If somehow you know the source, we can specify the origin. This will turn your environment much more security but be aware that this may create some trouble if you have differents source and eventually you forgot something.

Here we have a filter for HTTP traffic. In our Lab we are going to allows HTTP, SSH and Ping.

You don't need to worry about the Outbound Rules. Everything will be allowed in Outbound by default.

Our VPC is ready to receive hosts and to do that we are going to go to EC2.

On EC2, if this is the first time you are here, you will see that "Running Instance" will be zero.
What we are going to do now is create two new EC2 instances. One of them will be our public server and the other one our private.
 Take care by selecting your server. Depends on the kind of EC2 you select you´ll be charged. It is a good idea delete your EC2 after your Lab finished.

This said, hit "Launch Instance". I´m not teaching how to create an EC2 instance step-by-step here as this Post is too long already. But you can find this information easily on the Internet. If you find some difficulty, please let me know and I´ll help you.

My first Instance is up and running. Some important information is highlighted in yellow. We can see that the AZ (Availability Zone) is sa-east-1a, which I choose as DMZ and it has been given to our server a Public IP address of 56.67.206.38.

I am able to ping it, as I have allowed ICMP on my Security Group.

And we are able to access the server using SSH as well.

Until this point, our DMZ is ready. We have a Server on it and this server is reachable from the Internet.
Now, as our final step, we are going to create a EC2 on our Private network and see which is the difference:


As we said before, we can have as many Security Group as we need. Then, for our private server we are going to create a new Security Group. We are going to call it DB-SEC and we´ll allow ICMP and SSH.

 Notice that now I have changed our Source. As this server is going to be installed in the Private network, it will be only accessible through our public network. As I know the origin, I can allow only this network.

As we can see from the image above, this second server doesn´t have an public IP address. This happen because we have put it on the Private network sa-east-1c. But we can see that it has a Private IP address 10.0.2.88.

 Of course this IP will not ne accessible from the Internet, but we do can access it from our Public Server:

That´s it guys. If you, like me, are preparing for the AWS Architect Associate exam, let me know. We can exchange experience and help each other to clear the exam.

Cheers!!!

Login in a Amazon EC2 Server Using Certificate

 Today´s post is going to be about Login into a Amazon EC2 Server using Certificate. This is pretty simple. Fist of all, download the Certificate file from your instance:

Give it a name:

This procedure is going to create a .pem Key:

Use Putty Generator to load and generate your private Key. It will be a ppk file:

Save it and give it a name:

After that, you need to create a new SSH Sesseion on Putty:

 In the SSH option, find your .ppk Key:

  Map the .ppk file in your machine and don´t click Open yet. Rollout up to Sesseion again andd click "Save".

   Then, click Open:

Congrats, you have just login in a Amazon Server using Certificate.

Google Drive on CentOS

Today information's place is the Cloud. Every thing looks like to be sending to the Cloud. Although Cloud is just someone´s else computer, actually it is a great improvement in many sense.
  Recently my notebook´s HD just gone! I simply lost everything I had. I was working in a bunch of scripts and simply lost everything. I had no backup!.
 Well, based on this disaster, I decide to write this post. Here we are going to learn how integrade CentOS 2.6.32-279.el6.i686  with Google Drive and, from Command line interface, upload and/or download files. Google allows for 18 Gigabytes with no charge, which is a lot of space and we have no reason to do not use it.
 Considering you have a perfectly installed and configure CentOS CentOS-6.3-x86, we can keep going:
 The first thing to do is download the bynary from Google. Currently, the URL https://github.com/prasmussen/gdrive#downloads.
is ok. The option gdrive-linux-386 is the correct for this Operational System.
 You can right click and get the URL, then, use wget to donwload:

wget -O drive https://docs.google.com/uc?id=0B3X9GlR6EmbnLV92dHBpTkFhTEU&export=download


  The -O option permits you to change the downloaded file name, otherwise, the file names will look like a string. After that, copy file to /usr/sbin and give it permission:

mv drive /usr/sbin/drive  

chmod 755 /usr/sbin/drive 

 Now you have Google drive installed on you system and we have one more step to do:

 The command used here was "drive about". This command is for an Authentication step which is pretty simple. It gives you an URL and as bellow to enter a Code. As soon as you copy and paste that URL in a Browser, you´ll be asked to permit the operation:

By doing so, you´ll be presented to another page where a string of characters will be presented:

Copy that string and paste on the command line:

After that, the authentication process is finished:

You can now happily send your file to the Cloud and get those back according to your wishes.
 To upload a file just type "drive upload 'file name'" and it´s done. Simple like that.
 To download a file, one step more is necessary. You can´t just type "drive download 'file name'" because you are looking for something out there on the Internet. You need to pass out a reference.

The way I do that is:

By clicking with the right button over the file you want do download, you can see the URL for that file. 

As we can see from the image above, the file´s URL have a string id and this string can be used as a parameter on the download command. Just type "drive download 'string'"and you are ready to download a file from Google drive to you machine.
 You can also use de command "drive list". This command will show you all files currently on your Google drive and the string for each file



This is it. Pretty simple but really helpful. Now you can backup your files into the Cloud.