Test Systems: How to Protect Your Highly Sensitive Data

Test systems are the heart of your overall system design. They represent much of the history of your team’s code and development practices, which can include the beginnings of tools, applications, and services you provide to your users. Test systems serve as a record of how your systems, tools, and applications have matured and refined. Every genius and not-so-genius decision your team has made along the way can often be found there.

Test Systems: A Hidden Vulnerability

That said, the contents of test systems — test frameworks, CSV files, test data, etc. — can be ultra-sensitive. They could contain years upon years of private information, including intellectual property and competitive strategy. So, of course, you want to guard these with your life, right?

Well, yes. But often, in reality, that’s not what happens. Despite the sensitive nature of the data in these systems, they are often left unsecured or barely secured. This is because test systems are often built in an ad hoc way, and may even contain elements of ShadowOps. Often we find that they do not undergo the same security scrutiny that other parts of your development environment or production systems would. So the risk associated with them probably outstrips the regular investment that has been made to secure them.

What Constitutes Your Test Systems?

From a security point of view, it is time to toughen up the soft underbelly of your test systems so they don’t undermine your test, dev, and production environments.

First, let’s look at what we mean by test systems. Test systems generally constitute resources that are used by software testers (QA, SDET, SRE, customer service, product managers, etc.) to test and hone the products they want to release. These can include:

• Local virtualized servers (e.g., Docker, VMWare, VirtualBox).

• CI systems like Jenkins and the test resources connected to them.

• Demo or staging systems for manual testing or acceptance.

• Test databases for integration and load testing.

Non-unit testing resources are the most susceptible to intrusions because real resources are likely used more frequently than in-memory, mock resources.

Properly built test systems are generally situated inside isolated systems so that a system-under-test (SUT) can be controlled by the layer around it (e.g., a CI system like Jenkins or Travis CI, or language-specific test frameworks). Controlling this layer is important. So is preventing access to SUTs, the systems, and credentials. Building sandboxes is not quite enough, however. You’re going to need a real fence.

Best Practices for Protecting Test Systems

If you already integrate tools to manage your test systems and share them with an operations team’s configuration management tools or deployment process, then you can use the test systems to improve operations practices and tools while also improving the repeatability and documentation of the development and the accountability of the test systems. For example, you can use the same authentication tools in test and production instead of providing anonymous access. Anonymous access is an all-too-common practice and can lead to major security issues. This can, however, extend beyond authentication to most security practices.

Once you improve the security of your test systems, you can also help your development and QA teams by providing a safe, well-monitored environment in which you can:

• Test and deploy updated applications.

• Test applications to recently released security patches.

• Improve how your projects provide access to the outside world.

In other words, improved security improves the testability of your systems by providing another valuable perspective on the development and execution of your systems. Increasing the isolation of a system should always lead to increased security, knowledge, and testability of that system — goals that will help to unite the QA/SRE (site reliability engineering), operations, and security teams.

As you can see, protecting test systems is very important. It is critical that you follow defined policies and procedures to ensure that your test systems are as secure as possible and don’t expose existing vulnerabilities or create new ones.

Here are three main areas where you can improve the security of your test systems, along with specific recommendations for each.

1. Build a Perimeter

• Keep test resources inside a VPN and consider isolating test VMs in a VPN that is separate from production or demo systems. Ideally, a VPN containing test-specific resources should have no access to or interaction with development or production systems or their credentials.

• Require authenticated access to all test environments. Do not permit anonymous access. Anonymous access is only appropriate for well-developed local/local-virtualized testing against test data (i.e., non-real data).

• Control the security groups/firewall access of your test systems, and be as explicit as possible. All inbound and outbound traffic should be over known ports. Do not simply allow inbound and outbound access on all ranges; for test systems, be specific about what ports are open and what is communicating over them. This makes access specific and helps define the specifics of your system for QA at the same time.

• Monitor all access to and communication with the test system’s networks, and set up sane alert rules to ensure that your test systems remain properly configured. Do not disable your security perimeter by ignoring logging access and networking information for test systems. Be MORE specific about configuration in test systems, not less, especially when it comes to security.

2. Monitor and Alert on Test Systems

• Alert on platform actions such as those provided by AWS CloudTrail. Examples of activities to monitor include:

  • Backing up instances.

  • EBS volume copying and snapshotting.

  • Backing up test or development databases.

  • Execution of Lambda Services.

• Increase security alerting for login and communication events into test systems and networks. Assume that there should be no regular direct human access to automated test infrastructures. While no human should regularly be directly transferring files to or from automated test systems, your test and QA teams should be considered responsible and trusted to access these systems. However, they should still be monitored for any manual interaction.

• Increase security event logging in test systems, but be careful not to increase pages to on-call resources for test system failures. Alert fatigue does not help anyone reach any goal.

• Monitor ALL human (i.e., non-automated) behaviors/actions on test automation systems. These systems should be largely ephemeral and it should be uncommon that they have active user sessions.

3. Improve Your Tests, Improve Your Security

• NEVER use default credentials (ie: ‘postgres/postgres’). Default and root credentials are highly vulnerable targets, so disable them for any test system. Always run tests with known access credentials. This is especially true if you ever place production data on a test system to help debug major production issues (see the next bullet).

• Never use production data if you can help it; instead use generated, production-like data. Consider using database migration tools if your development teams are already using them, as they can improve the speed and reliability of loading and obfuscating test data.

• Shared/hosted test resources are more exposed than production systems if they are placed under identical scrutiny because of their common direct access and manually configured nature. Save any unauthenticated testing for your local test development, not for tests shipped to your CI pipeline for execution. These systems can often contain valuable information in the form of old, forgotten credentials. State actors are known to regularly breach systems and monitor actions and files on test systems to improve their ability to move laterally to production systems.

• Use secure methods to transfer sensitive configuration information to your test environments, and prevent plaintext secrets or passwords from being committed to source control or configuration management systems, where this information can easily be leaked. Use systems such as Travis CI’s encrypted environment variables or Jenkins’ credential store to dynamically inject variables into your build to reduce the possibility that secrets such as passwords are being checked into test code. This approach will also frequently improve the local development of tests and test infrastructure by making it easier to run tests in a variety of environments with diverse credentials, or even in local testing scenarios.

• Encrypt or secure test database and system backups as you would any production backup.

• Deprecate and remove any proof of concept test systems. There is a strong risk of credential leaking with any manually configured systems. If possible, move test systems into configuration management when graduating test systems out of the proof of concept, and ideally before a wide internal testing user base begins developing tests against a new test environment. Isolate ALL systems used to develop new test systems or patterns during development, and destroy them once more permanent testing resources are put in place (e.g., once tests have been integrated into a CI pipeline).

• Always patch your test systems. These are just as vulnerable as any other system, and you should not develop or release code to unpatched servers. Testing software that needs to run in known unpatched environments should be tested on hardware-isolated networks — which is not an ideal use case for the cloud. (Note: If you don’t think you can patch your production systems, stop reading this immediately. Then you can start learning how to patch and upgrade ALL your systems, platforms, and tools, and how to make it easier to patch, test, and deploy these changes to your production systems.) At the very least, you should begin using your test systems to help stage security patches to your production systems (test systems are strangely perfect for things like testing!).

• Do not test with plaintext credentials. Instead, use one-time or temporary credentials during test runtime if possible. The following services work extremely well to provide temporary credentials:

  • AWS Security Token

  • The Vault Project

Know Better, Do Better

When it comes to test systems, the lack of appropriate knowledge and/or lax security practices can cause these sensitive systems to become points of weakness or vulnerability. Following the recommendations outlined above, you should be able to toughen up the security of your test systems. Make it your goal to ensure that your team does not expose existing vulnerabilities or create new ones, and in doing so you will pass on the benefits of your test systems to your dev and ops environments.