Meli Proxy

Contents

• Introduction
• Features
• Installation
• Usage
• Configuration
• Testing
• Architecture Design
• Contributors
• License

Introduction

(go back to top)

This project serves as my solution to a development challenge in the context of an interview for a MercadoLibre position.

Here is a translated description of the task (original is in Spanish):

MercadoLibre currently runs its applications on more than 20,000 servers, and these typically communicate with each other through APIs, some of which are accessible externally (e.g., api.mercadolibre.com). One of the problems we face today is how to control and measure these interconnections. For this, we need to create and implement an "API proxy."

This proxy must meet at least the following requirements:

• Allow proxy functionality on the domain api.mercadolibre.com. Example: A request like curl 127.0.0.1:8080/categories/MLA97994 should return the content from https://api.mercadolibre.com/categories/MLA97994 (without redirect or caching).

• It should be possible to control the maximum number of requests (rate limiting), for example:

• Source IP 152.152.152.152: 1000 requests per minute.
• Path /categories/*: 10,000 requests per minute.
• Source IP 152.152.152.152 and path /items/*: 10 requests per minute.
• Other criteria or control alternatives are welcome.

• The average load handled by the proxy (as a solution) must be able to exceed 50,000 requests per second, so how the solution scales is very important.

• Usage statistics: Proxy usage statistics should be stored and (ideally) visualized. The statistics and control interface could support REST.

• Having all the features complete (and working) is ideal, but any level of completeness is acceptable.

• Having a diagram or some other visualization of the system's design, functionality, and scalability adds significant value.

• The solution should work against the real MercadoLibre API. However, some known issues with HTTPS may arise, so any alternative (mocks, another API, etc.) that tests functionality is also valid.`

Features

(go back to top)

The main functionality of this project is to proxy requests to a target API. The proxy will act as a logger to record all the requests made and as a WAF (Web Application Firewall) to control the amount of requests made by a client. The core elements of the project are 2 Golang services, the proxy and the manager.

Proxy features:

• Proxy requests to a target API (no cache, no redirect).
• Rate limiting based on the client's IP and the request path.
  • If WAF is triggered, the request will be blocked, and it will return a 403 status code.
  • Every request will be logged independently of the WAF status.
  • The logger is configured to log to stdout and, in the local version, to a RabbitMQ queue. The logger middleware can be easily extended to log to other services.
  • Log format will be a JSON object with the following fields:
    • datetime: The date and time of the request.
    • ip: The client's IP.
    • method: The HTTP method of the request.
    • path: The path of the request.
    • status: The status code of the response.
• The WAF rate limiter uses a sliding window log algorithm, with a window of 1 minute.
• WAF is configured to keep track of requests using an external cache, in this case, Redis.

This is an example image after limiting /pokemon/* to all IPs to 10 requests per minute and simulating a DOS attack.

Manager features:

(go back to top)

• The manager service is in charge of managing the WAF.
• It is an HTTP service that listens (by default) on port 8000.
• The service is REST-based, and we can check the specs of the routes in the OpenAPI Spec.
• We can limit the requests based on:
  • A specific client's IP: 192.168.0.1
  • A specific request path: /pokemon/ditto
  • A combination of both: 192.168.0.1 & /pokemon/ditto
  • Every IP: *
  • Every path: /*
  • A complete path tree: /pokemon/*
  • All the requests: * & /*
• We can rate the limit or completely block the requests. To do the latter, use 0 as the value.
• The manager and the proxy service will be connected through a cache, in this case, Redis.
• The manager will also write the rules to a NoSQL Database that can be synced with the cache in case of a restart (not implemented yet). Locally, we will use a MongoDB instance for this.

Note: Although we can limit based on a path tree, IPs should be exact or all. There is no support for wildcards or CIDR notation yet. Also, only IPV4 is supported on this demo.

Installation

(go back to top)

This project makes heavy use of Make and Docker, so please have them both installed on your computer:

• Make
• Docker

The core of the project consists of 2 Golang services, and for the local version of the design, we also use a Python consumer:

• Golang
• Python3

To develop and run some tests, you will also need Pipenv:

• Pip
• Pipenv

Usage

(go back to top)

The project relies almost entirely on Makefiles to run the different services and tests.

To see all available commands, run:

make help

To deploy the services locally, just run:

make

After running the make command, it will spin up the following services:

• proxy: The main service that will act as the proxy. Listen on port 8080.
• manager: The service that will manage the rate limiting. Listen on port 8000.
• consumer: A Python service that will consume the requests logs.
• redis: A Redis instance to store the connection data. Listen on port 6379.
• rabbitmq: A RabbitMQ Queue to send/receive the requests logs. Listen on port 5672.
• opensearch: An OpenSearch instance to index the requests logs. Listen on port 9200.
• opensearch-dashboards: A sidecar container for the OpenSearch service to visualize them and run queries easier.

Note: Ports are mapped to the host to allow access. Please change the defaults if you have them in use in your host machine. The how-to can be read in the Configuration section.

---

Once the services are running, you can start sending requests to the proxy service. In this case, the proxy will not forward the requests to the real MercadoLibre API, it will use a free API instead (PokeAPI). Please note that this is an external API, so latency will have a floor based on the API's response time. Also, please be aware when testing to not DOS their service. This can be changed in the docker-compose.yml file by changing the PROXY_TARGET environment variable.

Example

curl  http://localhost:8080/pokemon/1

Every request will generate a log that will be sent to the OpenSearch Dashboard. You can see them running:

make opensearch

Note: If this is the first time, you must configure the index pattern to requests* and the timestamp field to datetime. The Dashboard UI will guide you to do it anyway.

To stop the services, just run:

make stop

Configuration

(go back to top)

The services can be configured using environment variables. The following are available:

• Proxy:

  • PROXY_TARGET: The target API to proxy the requests. Default: https://pokeapi.co.
  • HTTP_PORT: The port where the proxy will listen. Default: 8080.
  • REDIS_HOST: The Redis URL to store the WAF data. Default: redis (docker container).
  • REDIS_PORT: The Redis port. Default: 6379.
  • RABBITMQ_HOST: The RabbitMQ URL to send the logs. Default: rabbitmq (docker container).
  • RABBITMQ_PORT: The RabbitMQ port. Default: 5672.
  • RABBITMQ_QUEUE: The RabbitMQ queue to send the logs. Default: logs.
  • USE_ENV_FILE: If set to 1, the proxy will read the environment variables from a .env file. Default: false.
  • ENV: The environment where the service is running, this will affect the env file to be loaded, example .env.test.

• Manager:

  • HTTP_PORT: The port where the manager will listen. Default: 8000.
  • REDIS_HOST: The Redis URL to store the WAF data. Default: redis (docker container).
  • REDIS_PORT: The Redis port. Default: 6379.
  • MONGO_HOST: The MongoDB URL to store the rules. Default: mongo (docker container).
  • MONGO_PORT: The MongoDB port. Default: 27017.
  • USE_ENV_FILE: If set to 1, the manager will read the environment variables from a .env file. Default: false.
  • ENV: The environment where the service is running, this will affect the env file to be loaded, example .env.test.

• Consumer:

  • RABBITMQ_HOST: The RabbitMQ URL to consume the logs from. Default: rabbitmq (docker container).
  • RABBITMQ_PORT: The RabbitMQ port. Default: 5672.
  • RABBITMQ_QUEUE: The RabbitMQ queue use. Default: logs.
  • OPENSEARCH_HOST: The OpenSearch URL to send the logs. Default: opensearch (docker container).
  • OPENSEARCH_PORT: The OpenSearch port. Default: 9200.

Testing

(go back to top)

There are two types of tests in this project:

• To execute the Unit tests run:

make test

This will run the tests for the proxy and manager services.

• To execute the Integration tests run:

make integration-test

This will spin up the services and run acceptance tests like an actual client.

Architecture Design

(go back to top)

Note: The following diagram is a high-level overview of the project's architecture. It was designed with a specific cloud provider (AWS) in mind but can be easily adapted to other providers. The local version intends to mimic the cloud version, following the general data flow. There are alternatives to every option, but I chose the ones that I thought were the most suitable for our requirements.

Let's break this down piece by piece:

• Network

The services run in a VPC (Virtual Private Cloud), with a minimum of 3 subnets, one public and two private. The VPC will have assigned an Internet gateway to allow the services to communicate with the outside world in case of need, and a NAT Gateway will be placed in the public subnet to allow the private subnets to communicate as well. Only one of the private subnets will be allowed to do so, the other subnet will have its route tables configured to not allow any traffic to the Internet, (neither inbound nor outbound). We will place our data services on this second subnet.

• Connectivity

We will host two zones in Route53, one public and one private. The public zone will be used to resolve a public domain to be able to operate the admin service from the outside (I did not know how the configuration was going to work, so if the admin is intended to be used internally, like a dynamic ad hoc rate limiter, we can dispose of this). This domain will point to a Load Balancer placed in the public subnet that will redirect the traffic to one of the Manager services instances (in the diagram, it is seen as a lone instance. I don't know how much the configuration will be used, so I thought that it did not need auto scaling, in case of need this can be adjusted). For the private zone, we will use it to resolve the internal services, like the proxy. We will point it to an internal Load Balancer hosted in the private subnet with the NAT route table (if the proxy should only redirect to internal services, all the NAT configuration can be torn down), and it will redirect the traffic to the Proxy instances. The proxy service is intended to be used internally, so I expect the other services to be in the same VPC (in case this is not always true, we can set up VPC peering to allow communication).

• Services

  • Proxy The proxy service will be deployed using ECS, pulling an image from a private ECR, and will be allowed to scale in/out as needed using the AutoScaling group feature EC2. The configuration of the instances, etc., can be tweaked to allow the service to scale to the 50k requests/second needed. The proxy will be placed in the private subnet with the NAT route table (again, if no outbound internet access is required, this will not matter).
  • Manager If needed, the manager can be configured to be deployed exactly like the proxy service or even changed to a Lambda serverless function. However, the proxy can not be that flexible due to the hard requests/second requirement (Lambda is not suitable for constant high traffic).
  • Consumer In this version, we will not use a consumer service or a queue. Instead, we will create a Kinesis Firehose service that will receive the logs from the proxy and send them directly to the OpenSearch service. We can hook up a lambda function to process the logs before sending them to the OpenSearch service if needed. The connection between the proxy service, Firehose, and the OpenSearch service will be made using VPC endpoints to avoid having traffic go through the Internet.
  • Opensearch The OpenSearch service will be a managed service hosted in the same VPC and will store the logs. Because it is in the VPC, the dashboard will be inaccessible from the Internet, so we will use a VPN to access it ( we can configure a Client VPN service or just use a bastion server with SSH Tunneling). Data access will be configured to only receive logs from the Firehose service, and the dashboard will be configured only to allow access from selected IAM users.
  • Redis For this solution, we will use an InMemory cache to process our requests as quickly as possible. In this architecture, we will use a managed ElastiCache Redis service hosted in the isolated private subnet. Traffic will be allowed from both the proxy and the manager services.
  • DocumentDB The manager service will use DocumentDB (a MongoDB API equivalent service) to store the rules and act as cold storage in case of need. This service will be hosted in the isolated subnet and accessed only by the manager service.

Contributors

(go back to top)

This project was developed by Ivan Monaco.

Contact can be made thru [contacto@ivancmonaco.com.ar](mailto: contacto@ivancmonaco.com.ar)

License

(go back to top)

Meli Proxy by Ivan Monaco is licensed under CC BY-NC-SA 4.0