Hosting an Angular 2 app with deep linking support in OpsWork (Chef 11 Stack)

The Problem

We recently developed an Angular 2 web application and after deploying it to a staging we encountered an issue with link sharing not working. The application was hosted on an AWS OpsWorks stack using the static web server layer.The static web server layer uses NGINX to serve the web application. The link sharing did not work as the links did not correspond to actual resources available on the server.

How do we support deep links ?

The knowledge of the link resource is available to the application router and thus NGINX needs to be instructed to route all traffic to the application root when a resource is not found. This can be achieved using the try_files directive [6] as follows:

try_files $uri$args $uri$args/ /index.html;

To test this we SSHed into the instance and accessed enabled sites configuration for the application. This is usually accessible at /etc/nginx/sites-enabled. The try_files  directive was added to the following block:

After making the change we restarted the NGINX service using the following command:

sudo service nginx restart

Working with OpsWorks

It worked! we were able to share links. However, we were now faced with the problem of adding this configuration to the OpsWork Stack configuration. The above configuration cannot be added using an attribute and instead must override a configuration template. The steps for doing this are listed here [3]:

1. Identify the cookbook that contain the configuration template. In this case it was the nginx cookbook which is located here [7].
2. Replicate the cookbook structure making sure only to keep the template that needs to be modified. 
3. Create a zip archive and upload it to a location which OpsWorks can access. In our case we used S3.

A cookbook with this updated template is available here [1].

Deploying to OpsWorks

1. Enable custom cookbooks for your OpsWork Stack
2. Provide the location of zip archive as the location of the custom cookbook
3. Execute a Run Command for the Update Cookbook action
4. Deploy the application

References
[1] https://github.com/splinter/angular2-deeplinks-opsworks-cookbook
[2] http://stackoverflow.com/questions/38991541/nginx-and-angular-2
[3] http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-template-override.html
[4] http://docs.aws.amazon.com/opsworks/latest/userguide/cookbooks-101-opsworks-templates.html
[5] https://serverfault.com/questions/378581/nginx-config-reload-without-downtime
[6] http://nginx.org/en/docs/http/ngx_http_core_module.html#try_files
[7] https://github.com/aws/opsworks-cookbooks/tree/release-chef-11.10/nginx
[8] https://github.com/aws/opsworks-cookbooks/blob/release-chef-11.10/nginx/templates/default/site.erb

CloudWatch DataAlreadyAcceptedException issue

The Problem:

We recently started using AWS Cloud Watch Agent for uploading logs for a few of our NodeJS processes. Soon after our initial move we began to notice that logs were not been uploaded for the majority of the processes.

Debugging Process:
In order to debug the issue further we began by looking at the CloudWatch agent logs located in the var/log directory.

We noticed the following logs in the /var/log/awslog.log:

2017-03-24 05:32:48,758 - cwlogs.push.publisher - WARNING - 14818 - Thread-5 - Caught exception: An error occurred (DataAlreadyAcceptedException) when calling the PutLogEvents operation: The given batch of log events has already been accepted. The next batch can be sent with sequenceToken: 49567766878003081255417777553615842777473923345314516562

The cause of DataAlreadyAcceptedException is a bit vague:

The event was already logged.

However, reading about the component which throws this exception gives an inkling of the cause [2]. The exception is thrown by the PutLogEvents API. The official documentation [1] lists the following conditions under which a log may fail to upload:

 - The maximum batch size is 1,048,576 bytes, and this size is calculated as the sum of all event messages in UTF-8, plus 26 bytes for each log event.
 - None of the log events in the batch can be more than 2 hours in the future.
 - None of the log events in the batch can be older than 14 days or the retention period of the log group.
 - The log events in the batch must be in chronological ordered by their timestamp (the time the event occurred, expressed as the number of milliseconds since Jan 1, 1970 00:00:00 UTC).
 - The maximum number of log events in a batch is 10,000.
 - A batch of log events in a single request cannot span more than 24 hours. Otherwise, the operation fails.

In our case logs which failed to upload met all criteria except the following:

 A batch of log events in a single request cannot span more than 24 hours. Otherwise, the operation fails.

Oddly enough this was preventing even current logs from been uploaded to CloudWatch.

Solution:

The state of the AWS CloudWatch agent is maintained in a file defined in the CloudWatch configuration file.

In our case the state file was defined in our cwlogs.cfg file was:

/var/awslogs/state/agent-state


In order to force the CloudWatch Agent to force upload the logs we deleted the state file.



References

[1] http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html
[2] http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html#API_PutLogEvents_Errors
[2] CloudWatch Agent Reference, http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html
[3] CloudWatch Agent start command, http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/StartTheCWLAgent.html
[4] CloudWatch Agent stop command, http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/StopTheCWLAgent.html
[5] CloudWatch Agent status command, http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/ReportCWLAgentStatus.html