Are you doing benchmark testing on databases? Here are two resources for large amounts of test data.
If you’re like me, at least one of your resolutions for 2013 is related to technology. Perhaps you want to learn a new language or contribute to open source software projects. One resolution that you should consider is to learn about Amazon Web Services and to manage your own server. Whether you’re in development, creative, or on the business-side, you’ll benefit from having at least a basic understanding of how to setup your own server in the cloud.
So what would you do with your server? Many businesses choose Amazon Web Services for hosting their production and testing application infrastructure. You may want to do that, but did you know that you can easily use it for a development environment?
Why would you use AWS for a development environment? Here are a few reasons:
- You use Windows or Mac OS on your local machine and you want to develop on the same platform as your production Linux environment
- You use a cutting edge OS (e.g., Ubuntu+1) but don’t want to continually setup your local development environment when things break
- You want to have access to your development environment from any device
- You’re an information architect, designer, or product manager (in other words, not an engineer) who wants to UAT branches before new code lands in QA but isn’t confident in modifying your local machine
The good news is that there are lots of resources available to help you use AWS. And even better news is that new AWS customers can run a free Amazon EC2 Micro Instance for a year! Even after the free year is up, monthly costs for the Micro Instance are very low.
I’ll begin by walking you through setting up your EC2 instance, installing a few key applications, and setting up an ssh key.
In a following post, I’ll describe how to access your EC2 instance from an Android or iOS device, use Virtual Environments (for Django) and run your server to view your application.
Let’s get started.
- Sign up with AWS
- Use the AWS Management Console to launch an EC2 instance. Christophe Coenraets’ post does a good job describing this process in steps 1 through 3. Amazon’s “Quick Launch Wizard” looks to be the fastest path to create a new instance. In this process you will create and download (as a .pem file) a new Key Pair and select the launch configuration. I recommend the most recent Ubuntu Server LTS but this depends on your testing and production environments.
By default, AWS will create the new instance using the “quicklaunch-1” Security Group. This means that only port 22 will be open on the machine so you can SSH into it. To open more ports (like port 80 for a web server), select “Security Groups” from the EC2 Dashboard and modify the appropriate group.
- Copy the .pem file to your ~/.ssh folder, update the file permissions (chmod 600 [filename]), and SSH into the EC2 instance using Terminal (in Linux or Mac OS) or PuTTY (in Windows).
The format for logging in to your EC2 instance is “ssh -i [path to .pem file]/[.pem filename] ubuntu@[public DNS]”. The public DNS is available from the Instances screen on the EC2 Dashboard or you can setup an Elastic IP and associate it with your EC2 instance.
- Update the Ubuntu package index and upgrade packages
- If your application requires Apache, MySQL, and PHP, install LAMP using the ‘sudo tasksel’ command and selecting “LAMP server”
- Install the distributed version control system in use by your engineering team (like Git or Bazaar)
Now you have an EC2 instance with a web, application, and database server running along with a version control system. This is a good point to take a snapshot so you can skip steps if you ever need to redeploy this instance.
Finally, you want to be able to pull code from your code repository (e.g., GitHub or Launchpad). In order to do this, you need to add your personal SSH key to your EC2 instance… but you should probably check with your engineering team lead on what’s required to get your application’s code and any branches. You can copy your key from your local machine or create a new key from the running EC2 instance.
And some final notes.
- Get to know VIM for editing text files on your server. There’s a bit of a learning curve but you’ll pick it up quickly.
- If you need to use multiple SSH keys, an SSH config file will make life a lot easier. There are many great resources available describing how to create one.
- For help managing your Ubuntu Server, visit AskUbuntu.com. The Ubuntu community is always willing to help others.
Part 2 coming soon.
Getting ready for a morning walk to the doughnut shop.
I’ve worked with engineering teams that have used a variety of methodologies. Here are some tips on a recent approach that worked quite well. It’s an agile/lean/kanban approach that enabled me (the product manager) to work closely with engineering, can be deployed quickly, and minimized waste. This is a supplement to a brief presentation.
1. Above all…
Reduce barriers as much as possible in team communication, development environment setup, branching application code, testing, and deployment.
Measure the time it takes for an engineer to setup their environment. If it takes days or even hours, you’re missing opportunities to reduce waste. Putting in the effort early to get this process down to a few commands that can be executed within minutes frees up valuable time in the future for developing new products and improving the quality of your application.
Along with an efficient environment setup process, engineering teams should always write tests. These are valuable tools that will help you catch bugs early and save time at all stages of the deployment process.
2. Do code reviews
Consider reviews to be the gate that code must pass through in order to be merged. Keep it simple – approve or reject – and encourage reviewers to give feedback in all scenarios. Code reviews are an excellent opportunity for everyone to learn about the new code entering the application but also suggest ideas on how to make it better.
For teams that are doing code reviews, limiting branches to a max of x lines (like 500) has a few benefits.
- Encourages engineers to land code regularly instead of once a week or longer
- Forces people to think more about how to split up the work and ship value more frequently
- Trains the other engineers across projects as they review merge proposals in small chunks
3. Open communication
While email has some benefits as a communications medium, it can really hurt an agile team.
- It’s not real-time
- Can easily lead to silos of information
- Becomes another system to manage access
- Open bug trackers empower anyone on the team or in the company to report issues. Bug reviewers should reply with honest feedback on the bug’s status and progress toward resolution (or “won’t fix”).
- Distributed Version Control Systems (DVCS) like Git, Mercurial, and Bazaar are a key component to transparent development. Use them.
- Wikis, text chat (e.g., IRC, HipChat), and even video conferencing (e.g., Google Hangouts) and screensharing from Google Apps helps teams collaborate openly and multi-task whether they’re in the same office or on a different continent.
- Finally, tools that simply share basic text like pastebin, gist, and etherpad can save time and enhance collaboration.
4. Continuous integration
Invest in continuous integration (e.g., Jenkins). It will make your operations more efficient, shine a spotlight on problems earlier, and save you a lot of pain in the long run. Use something like Selenium to constantly test your application automatically and raise flags when failures occur. Ensure all team members can stop the line and fix the problem when important issues are identified.
Use multiple environments in your architecture for testing and getting feedback on enhancements. A staging environment enables testing of enhancements prior to a production release. For this environment, using a copy of the production database will give you more accurate test results without the risk of damaging live user data. An edge environment using the production database and accessible to everyone is also useful to test new features before they’re finalized.
Frameworks are wonderful tools that help your engineering teams move fast. Unfortunately, making poor choices (e.g., using multiple frameworks that deliver the same functionality) can result in higher long-term maintenance and support costs and slowly chip away at the efficiency of your team. So make careful decisions to keep your team agile and contribute any enhancements upstream.
7. Final bits
Do daily standups because it’s a fast way to keep everyone on the team in the loop. Keep it to 15 minutes. Each person should share what they did and what they’re going to do next.
Periodically look at your development processes. Waste is sure to creep in so identifying and removing anything that isn’t directly adding value is a good thing to do every now and then.
Have a regular cadence for your releases and target each piece of work to those releases. Having a defined schedule will help to keep the entire product team (including marketing and sales) stay in the know about when features will be released.
I hope this will help your teams to be more agile and enhance the development of your extraordinary products. Best of luck!
Wildebeest at the beach house.