Debugging user data scripts on Amazon EC2 is a bit awkward indeed, as there is usually no way to actively hook into the process, so one ideally would like to gain Real time access to user-data script output as summarized in Eric Hammond's article Logging user-data Script Output on EC2 Instances:
The recent Ubuntu AMIs still send user-data script to the console
output, so you can view it remotely, but it is no longer available in
syslog on the instance. The console output is only updated a few
minutes after the instance boots, reboots, or terminates, which forces
you to wait to see the output of the user-data script as well as not
capturing output that might come out after the snapshot.
Depending on your setup you might want to ship the logs to a remote logging facility like Loggly right away, but getting this installed early enough can obviously be kind of a chicken/egg problem (though it works great if the AMI happens to be configured like so already).
It might also be useful for you to see what the userdata looks like when it's being executed during the bootstrapping of the instance. This is especially true if you are passing in environmental variables or flags from the CloudFormation template. You can see how the UserData is being executed in two different ways:
1. From within the instance:
# Get instance ID
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
# Print user data
sudo cat /var/lib/cloud/instances/$INSTANCE_ID/user-data.txt
2. From outside the instance
Note: this will only work if you have configured the UserData shell in such a way that it will output the commands it runs.
For bash, you can do this like as follows:
"#!/bin/bash\n",
"set -x\n",
Right click on the EC2 instance from the EC2 console -> Monitor and Troubleshoot -> Get system log. Download the log file and look for something a section that looks like this: