05 Jun 2020

Phase 3: Building a Site Visit Counter with AWS API Gateway / Lambda / DynamoDB

Mohamed Muhannad

In this part of the challenge, we’re supposed to build a visit counter feature for our static website.

Honestly, this may have been the hardest part of this challenge so far. My only exposure to software development has been scripting. That is something which is very procedural and once off. But the lack of actual software development background meant I had to really rack my brain until things clicked. In addition to building a Lambda Function using Python, I had to figure out how to put it between API Gateway and DynamoDB. I had never used these services at all and was clueless for a good while.

When faced with new things, I usually try out the examples in the official documentation to get an idea. AWS has good documentation with guided examples. But sometimes they can be too basic, and only explain just one part of a feature. It’s up to you to figure out how it fits into the overall solution. So I went back to basics, and took a beginner bootcamp course on building Serverless applications.

Lessons Learned

Plan Ahead

As a beginner to software development, I found that it’s good to plan out what you’re trying to build. It’s exciting to start the actual work and get clacking away at the keyboard. But having a plan helps keep you on track and not deviate when trying to code out different parts of the application.

A diagram is extremely useful.

I started out with what data should be saved in DynamoDB.

website # url of the website
counter # visit count
last_updated # timestamp when counter was last updated

Next I thought of how the website front-end would access the API to retrieve and update the data. I came up with the scheme below for API Gateway.

/api/visit -> # sending a POST request will update counter data and return the updated value.
/api/visit/<url; example.com> -> # sending GET request will fetch recent data.

Now for the Lambda function which is the glue to both pieces I went and mapped out how it would operate with the database and requests sent from API Gateway.

Functions to handle the database operations

# for this part boto3 package for Python is required
-> database connection
-> create table
-> put item
-> get item
-> update item

Functions to handle the api requests

-> what to do if GET method
-> what to do if POST method
-> what to do if neither method

Coding

Using functions to manage common tasks.
Using try catch methods to handle errors.
Structured responses, so that front end can utilize them easily.
Know when to stop tinkering. I may have already got the code to work, but trying out how someone else on StackOverflow has implemented it would lead me down a rabbit hole. Before I know it, I’d be more confused.

The key takeaway here is that I need to practice everyday. Also study more on object-oriented concepts. I feel like I can improve the application more using that approach.

Javascript

It is an easy language to pick up. But there’s still lots more to learn. Things like async, promises and await are going over my head at the moment 😓. More experimenting is needed to understand the concept better.

DynamoDB

Quickest way to setup up a local DynamoDB is using Docker.

docker pull docker pull amazon/dynamodb-local
docker run --rm -d -p 8000:8000 --name dynamodblocal docker pull amazon/dynamodb-local

Lambda

Understand mixed use of boto3’s client vs resource methods. (I found that both options are for low level or high level operations DynamoDB resource. High-level is more suited for data manipulation.)
Make use of the test events before integrating with API Gateway. This would save lot of time instead of going back and forth between the 2 services

API Gateway

Use of Lambda Proxy Integration. This means that all requests from the API Gateway will be proxied to the Lambda function. So the function must be able to process them and return appropriate responses. (eg: CORS headers, response codes, messages, validation etc.)
Got bit confused with the python dict and json types of response when passing from Lambda function to API Gateway.

Writing Tests

I had only heard of testing code but never saw it in action. This was my first attempt at such a topic. Some Python specific and general testing ideas are:

Used unittest.
Tests written for Lambda function logic and database operation functions.
Use of setupclass and tearDownClass to setup a test database and not for all tests (like in setUp).
There seems to be mixed views on running tests for database operations. At this time, I’m testing against a local database. This wouldn’t work on a CI pipeline if a test database is missing. People have been recommending using mock tests, so I might need to modify the test code.

Conclusion

We really take all these web apps we use daily for granted. From this part of the challenge alone, I realized the huge number of moving parts there are to something that we so easily consume everyday. Kudos to all you software devs! 👏 I aspire to join your ranks one of these days.

Overall, I am so happy and relieved to be able to complete this part. There were times it felt like this is too much work and I’d never understand this. But the key here is to keep at it consistently. If it’s overwhelming, try to break it down into smaller pieces. That way you can focus on just that and not think of other parts of the application. Worst case, drop the coding and take a break to reset your mind (have you turned it off and on again?) 😄

Also practice coding everyday, even a small bit. That really helps solidify what you have learned and you get more confident in writing code.