Skip to main content

Posts

What is the Elastic Stack and how to post data to an ElasticSearch DB in an Amazon ES Service

Amazon ES Service is a fully managed system that makes it easy to deploy Elastic Stack to AWS servers in an integrated way. Some features like installing Kibana plugins are not yet available. ElasticSearch is part of the Elastic Stack, a group of tools/services from the Elastic Company (elastic.co) Elastic Stack: * Kibana * ElasticSearch * Beats * Logstash ElasticSearch is a NoSQL document database and is most common used with Kibana , a UI tool to visualize data from the ES database. ElasticSearch is used for high speed text search. It was previously known as the ELK Stack because of the tools/services from Elastic Company that are used togheter: * ElasticSearch * Logstash * Kibana In elasticsearch db you post data to an index in the same way you insert data into tables in a RDBMS (Relational Database Management Systems). We use index to separate and group different types of information (data) in the same way we use tables in a database. SO INDEXES ARE FOR ES...

API GATEWAY: THE CLOUDFRONT 403 FORBIDDEN ERROR

If you are having a 403 Forbidden error from CloudFront , that means your domain name is not linked to your CloudFront distribution and because CloudFront stays in front of your API GATEWAY you need to create a CNAME record pointing your domain name to your CloudFront target domain name in order for it to work. So, If you need to point your api to a custom domain name, all you have to do is following those 2 easy steps: 1 - CREATING YOUR CUSTOM DOMAIN NAME Go to the API GATEWAY console and click on the Custom Domain Name menu. Click on the Create Custom Domain Name button. Next, assign a certificate matching the same domain name you are creating and map to the root path and destination of your desired api. Lastly, copy the CloudFront Target Domain Name . You will need to paste that in your Route 53 record. 2 - CREATING A CNAME RECORD ON ROUTE 53 Create a CNAME record on Route 53 for the same custom domain name, assigning to it the CloudFront Target Domain. CONCLUSION ...

Getting to know the S3 Event Object

In this post i am going to show you how you can get into the content of the file that trigged your lambda. As we have seen in the previous post, the event object is passed as an argument to our lambda function, like this: exports.handler = (event, context, callback) => { }; When you place a trigger on a Lambda function based on a S3 event, S3 will pass all the context information that you need inside of the event object. In this case, our lambda will be trigged by an S3 event and the event object that will be passed to it will have a Record property with an array of objects inside of it like is shown below: { "event":{ "Records":[ { "eventVersion":"2.0", "eventSource":"aws:s3", "awsRegion":"us-east-1", "eventTime":"2018-09-22T14:25:20.411Z", "eventName":"ObjectCreated:Put...

Getting to know the Lambda Event Object

With any kind of http request, being it a simple web server or an api gateway, we will usually need to log different kind of information regarding the received request. But how can we do that with Lambda and how can we get all available information from it? Thats when the event object comes in place. The API GATEWAY will send all information about the request encapsulated in the event object passing it as an argument to our lambda function, like this: exports.handler = (event, context, callback) => { }; The event object have the following nested objects as shown below: 1. resource 2. path 3. httpMethod 4. headers 5. multiValueHeaders 6. queryStringParameters 7. multiValueQueryStringParameters 8. pathParameters 9. stageVariables 10. requestContext 11. body 12. isBase64Encoded { "resource": "/", "path": "/", "httpMethod": "GET", "headers": {}, "multiValueHe...

API Gateway Integration with Lambda : How to get all querystrings params

If you stringify the event you will notice that the queryStringParameters is actually an object rather than an array. So in order to get all queryStrings params passed in an API Gateway request you can simply make a for each key value pair in the object like this: var element = {}; for (var key in event.queryStringParameters) { element[key] = event.queryStringParameters[key]; }

How to move sql file into RDS

In this post, we are going to see how to bring data from mysql to RDS. The first thing you gonna need to do is a dump of your db to a sql file so we can than bring it into the RDS. So, here we go. If you have not done it already go to the RDS console and create or launch a new mysql db instance. Take note of the following infos: db name, db user, db passwd and the db endpoint. We gonna use all that with mysql command to bring all tables of our mysql into the RDS. So, next we need to create our dump file. Log into your instance to get access to mysql cli. You can login like this: ssh -i file.pem user@IP To get inside mysql cli: mysql -u root Create a dump of your database: mysqldump -u root db_name > file.sql This will create a dump of your db on your user location. Exit the mysql cli to check if the file was created. We can take advantage of the mysql cli of our previously db to run our last command to bring the file into our new RDS instance. mysql -h yo...

Changing AWS API GATEWAY Resource Name - Using the CLI

One of these days i had to rename a resource name on the AWS API GATEWAY, so here are the steps needed. In order to change your resource name you need to run 3 commands on the cli. One to get your API ID, one to get the Resource ID and one last command to change its name. First lets gonna find the ID of the desired API. Using the aws cli type the following command: aws apigateway get-rest-apis The API will return a json with an array of items containing all of your apis. Look into the name property to find the api that have the resource you need to change. Right above the name property you have the id property of your api. Take note of this id and run the following command with it to get the resource id. aws apigateway get-resources --rest-api-id 1010101010 Get the id matching the resource path that you are looking for to run the final command thats going to change your resource name. aws apigateway update-rest-api --rest-api-id 1010101010 --patch-operations op=replace,pat...

Copy a file to a remote instance on AWS using SCP

Many times you gonna need to move a file from your local computer to a remote server or download a file from a remote instance just using the terminal. You can achieve that by using the scp command: scp -r -i file.pem ./file.csv ubuntu@IP:downloads Notice that your destination folder is specified the IP Address. In this case, the downlod folder in the remote server. If you need to copy/download a file from the remote server to your local machine, just have to switch scp -r -i file.pem ubuntu@IP:downloads/file.csv . The dot (.) being your current directory in linux.

Copy CSV file to Postgres using \copy

Log into your virtual instance. In my case, i will ssh into an ec2 instance using a pem file like this: ssh -i file.pem ubuntu@IP Next, log into the postgres client followed by your password. psql -U user -h hostname dbname Postgres make use of the COPY command to deal with large volume of data more efficiently. \copy table_name(field1,field2) FROM path_to_file_on_server.csv DELIMITER ';' When using the \copy command we do not use the quotes on the file name. A path to a csv file on the server could look like the following: /home/ubuntu/downloads/file.csv