Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

Thursday, December 18, 2014

AWS OpsWorks FAQ and common commands

To create a deploy via CLI

Before using the command, you will need to first setup AWS cli on your computer, use the official AWS guide here

$ aws opsworks --region us-east-1 create-deployment --stack-id 9xxxx3-xxxx-455f-bc5c-8xxxx7a --app-id 090xxx88-2d32-xxxx-a460-faaxxxd7 --command "{ \"Name\": \"deploy\" }"

Note: AWS OpsWorks supports only one endpoint, opsworks.us-east-1.amazonaws.com (HTTPS), so you must connect to that endpoint. You can then use the API to direct AWS OpsWorks to create stacks in any AWS Region. Source

The following message will be returned if command issued accepted.
----------------------------------------------------------
|                    CreateDeployment                    |
+---------------+----------------------------------------+
|  DeploymentId |  912b37aa-f423-4dda-9966-021680bbc1a8  |

+---------------+----------------------------------------+

===================================================================================
To Create and Schedule a time-based instance via CLI

Below sample command will create a time-based instance, stack-id, layers-ids, instance type are mondoroty

aws opsworks --region us-east-1 create-instance --stack-id 58af7c69-xxxx-xxxx-8e6a-7660d7a18e55 --layer-ids 6e876603-ac96-4714-89a0-2453e1dc06f0 --os Custom --ami-id ami-6d1b343f --instance-type t2.medium --subnet-id subnet-1ae9f56e --auto-scaling-type timer --ssh-key-name sysadmin

Next, You will need to assign scheduler to wake or shut instance off, here's the sample command. The command will schedule instance 064c0869-xxxx-xxxx-9728-53d271c4a484 to power on between 00800 to 1900GMT+8 from Monday to Friday. Saturday and Sunday is included in the command to override any schedule on previously set to ON

I personally find that the command is not so user friendly as you will need to include each and every hour slot into the command.


aws opsworks  set-time-based-auto-scaling --region us-east-1 --instance-id "064c0869-xxxx-xxxx-9728-53d271c4a484" --auto-scaling-schedule "{\"Monday\": {\"0\": \"on\", \"1\": \"on\", \"2\": \"on\", \"3\": \"on\", \"4\": \"on\", \"5\": \"on\", \"6\": \"on\", \"7\": \"on\", \"8\": \"on\", \"9\": \"on\", \"10\": \"on\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}, \"Tuesday\": {\"0\": \"on\", \"1\": \"on\", \"2\": \"on\", \"3\": \"on\", \"4\": \"on\", \"5\": \"on\", \"6\": \"on\", \"7\": \"on\", \"8\": \"on\", \"9\": \"on\", \"10\": \"on\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}, \"Wednesday\": {\"0\": \"on\", \"1\": \"on\", \"2\": \"on\", \"3\": \"on\", \"4\": \"on\", \"5\": \"on\", \"6\": \"on\", \"7\": \"on\", \"8\": \"on\", \"9\": \"on\", \"10\": \"on\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}, \"Thursday\": {\"0\": \"on\", \"1\": \"on\", \"2\": \"on\", \"3\": \"on\", \"4\": \"on\", \"5\": \"on\", \"6\": \"on\", \"7\": \"on\", \"8\": \"on\", \"9\": \"on\", \"10\": \"on\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}, \"Friday\": {\"0\": \"on\", \"1\": \"on\", \"2\": \"on\", \"3\": \"on\", \"4\": \"on\", \"5\": \"on\", \"6\": \"on\", \"7\": \"on\", \"8\": \"on\", \"9\": \"on\", \"10\": \"on\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}, \"Saturday\": {\"0\": \"off\", \"1\": \"off\", \"2\": \"off\", \"3\": \"off\", \"4\": \"off\", \"5\": \"off\", \"6\": \"off\", \"7\": \"off\", \"8\": \"off\", \"9\": \"off\", \"10\": \"off\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}, \"Sunday\": {\"0\": \"off\", \"1\": \"off\", \"2\": \"off\", \"3\": \"off\", \"4\": \"off\", \"5\": \"off\", \"6\": \"off\", \"7\": \"off\", \"8\": \"off\", \"9\": \"off\", \"10\": \"off\", \"11\": \"off\", \"12\": \"off\", \"13\": \"off\", \"14\": \"off\", \"15\": \"off\", \"16\": \"off\", \"17\": \"off\", \"18\": \"off\", \"19\": \"off\", \"20\": \"off\", \"21\": \"off\", \"22\": \"off\", \"23\": \"off\"}}"


Command to show instance detail
aws opsworks --region us-east-1 describe-time-based-auto-scaling --instance-id 398913d7-ea3e-xxxx-xxxx-44ffc8befa46

Output:-
-------------------------------------------------------------------------------------
|                           DescribeTimeBasedAutoScaling                            |
+-----------------------------------------------------------------------------------+
||                       TimeBasedAutoScalingConfigurations                        ||
|+---------------------------------------------------------------------------------+|
||                                   InstanceId                                    ||
|+---------------------------------------------------------------------------------+|
||  398913d7-ea3e-4c91-9b16-44ffc8befa46                                           ||
|+---------------------------------------------------------------------------------+|
|||                              AutoScalingSchedule                              |||
||+-------------------------------------------------------------------------------+||
||||                                   Friday                                    ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  on |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on   |  on   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||                                   Monday                                    ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  on |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on   |  on   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||                                  Saturday                                   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  off|  off |  off |  off |  off |  off |  off |  off |  off |  off  |  off  ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||                                   Sunday                                    ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  off|  off |  off |  off |  off |  off |  off |  off |  off |  off  |  off  ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||                                  Thursday                                   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  on |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on   |  on   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||                                   Tuesday                                   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  on |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on   |  on   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||                                  Wednesday                                  ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  0  |  1   | 10   |  2   |  3   |  4   |  5   |  6   |  7   |   8   |   9   ||||
|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||
||||  on |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on  |  on   |  on   ||||

|||+-----+------+------+------+------+------+------+------+------+-------+-------+|||

http://docs.aws.amazon.com/cli/latest/reference/opsworks/set-time-based-auto-scaling.html

Friday, November 22, 2013

Install S3fs on Amazon Linux/Centos


S3f3 is FUSE-based file system backed by Amazon S3 which you can mount your S3 bucket to Linux machine.

Use cases:-
- Turn your backup folder to unlimited storage pool
- Serve as centralized media storage location for multiple servers.

Limitation:-
  • Objects size can only up to of 5GB.
  • You can't update part of an object. If you want to update 1 byte in a 1GB object you'll have to reupload the entire file.
Here's the installation steps:-
sudo yum groupinstall "Development Tools";
sudo yum install curl-devel libxml2-devel openssl-devel mailcap

cd ~;
wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.9.3/fuse-2.9.3.tar.gz;
tar -xzvf fuse-2.9.3.tar.gz;
cd fuse-2.9.3;

./configure --prefix=/usr;
make;
sudo make install;
sudo ldconfig;
export PKG_CONFIG_PATH=/usr/lib/pkgconfig;
#Verify version
#pkg-config --modversion fuse

cd~;
wget http://s3fs.googlecode.com/files/s3fs-1.73.tar.gz;
tar -xzvf s3fs-1.73.tar.gz;
cd s3fs-1.73;
./configure --prefix=/usr;
make;
sudo make install;


vi /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs


mount s3fs [bucketname] ~/s3bucket
umount: fusermount -u ~/s3bucket

mount on boot
s3fs#s3bucket /mnt/s3bucket fuse allow_other,use_cache=/tmp 0 0


Sources: http://kisdigital.wordpress.com/2011/08/04/installing-s3fs-on-rhelcentos/
http://www.idevelopment.info/data/AWS/AWS_Tips/AWS_Management/AWS_20.shtml
http://www.turnkeylinux.org/blog/exploring-s3-based-filesystems-s3fs-and-s3backer

S3fs Home
Fuse Home

Tuesday, April 9, 2013

AWS - S3 - Apply bucket policy for public read and office IP read and write


Here's the sample S3 Bucket policy when you have a pulic read bucket but only restrict write/update access to office network

If you enable everyone list your bucket from permission menu, everyone could grep the whole list of our bucket object by browsing your root domain url 



 {
  "Id": "Policy1346919974114",
  "Statement": [
    {
      "Sid": "Stmt1346917860156",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::origin-pdf.domain.com/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "110.174.240.29/26",
            "175.143.152.282/32"
          ]
        }
      },
      "Principal": {
        "AWS": [
          "*"
        ]
      }
    },
    {
      "Sid": "Stmt1346919900506",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::origin-pdf.domain.com/*",
      "Principal": {
        "AWS": [
          "*"
        ]
      }
    }
  ]
}