The newly released AWS Secrets Manager Agent is part of AWS’s ongoing effort to simplify the way developers and applications securely manage sensitive information such as API keys, DB credentials, and encryption keys. Traditionally, interacting with AWS Secrets Manager has required integrating AWS SDKs directly into your applications, or leveraging custom solutions like Lambda extension layers, CSI Providers, native integration with Amazon ECS, or vended caching libraries. While these approaches are effective, they add complexity and potential overhead to the development process.

To alleviate this, AWS introduced the Secrets Manager Agent, designed to streamline secrets management and minimize the burden on developers. However, while the agent simplifies secret retrieval, it introduces a new set of trade-offs between achieving high availability and securing the in-memory cache that can significantly impact security if not properly managed. For security practitioners, it's critical to understand both the potential benefits and the risks that this agent can introduce into production environments, as well as how to proactively manage security responsibilities when adopting this new tool, as we’ll explore in this blog.

What is the AWS Secrets Manager Agent?

The AWS Secrets Manager Agent is an open-source tool, developed in Rust, making it cross-platform and able to run natively across different environments. Its purpose is to help developers easily retrieve secrets (like API keys, passwords, and tokens) from the AWS Secrets Manager without relying on AWS SDKs.

This agent can be deployed as a sidecar in a containerized environment or as a parallel process to the main application. Applications interact with the agent by sending a simple HTTP request, including a known secret as a header (using X-Aws-Parameters-Secrets-Token), and the agent retrieves the necessary secrets.

Advantages of the AWS Secrets Manager Agent

  1. Simplifies Secrets Retrieval: The agent abstracts the complexity of interacting with AWS SDKs, providing a streamlined approach to secret retrieval.
  2. Reduces Application Size: By removing the AWS SDK dependency, the agent can reduce the overall size of containerized applications.
  3. Built-in Caching: The agent offers an in-memory caching mechanism for secrets, reducing latency for frequently accessed secrets.

The Hidden Risks: What You Need to Know

Despite these advantages, let us look at the customer side of responsibility to deploy and use the AWS Secrets Manager Agent in a secure way.

1. Lack of HTTPS Support

The agent communicates over HTTP (rather than HTTPS), which introduces a serious risk in environments where network configurations are not tightly controlled. In misconfigured environments—such as a shared network mode where multiple containers share the same namespace—an attacker could potentially sniff traffic and steal secrets being transmitted by the agent. The lack of encryption is a glaring vulnerability that can lead to the exposure of sensitive data: the access to secrets from the in-memory cache is controlled by the default SSRF token which is controlled by user file permissions. If the user permissions are not appropriately controlled, it can lead to the potential exposure of secrets to other applications running on the same computer that do not require access.

2. Single Point of Failure: Symmetric Authentication Token

While there is an IAM auth needed to fetch the secrets inside the agent, but once the secret is cached, the agent uses a symmetric key (the SSRF token). If this token is not rotated frequently, the system becomes vulnerable to the “one secret to rule them all” scenario. In this case, compromising the SSRF token can give attackers access to all the secrets the agent is authorized to retrieve.

3. Misconfigured IAM Roles

One risk associated with using the AWS Secrets Manager Agent is its support for IAM user credentials (Access Key and Secret Key) to access Secrets Manager. This risk also applies to the AWS SDK approach, as both methods rely on the same authentication mechanism. The core issue is the use of static credentials, which poses a threat if those keys are compromised. In both cases—whether using the AWS SDK or the AWS Secrets Manager Agent—the recommended best practice is to use IAM role assumption. By doing so, the agent (or SDK) can leverage the role attached to the instance or container, significantly reducing the exposure to long-lived credentials.

4. Caching Vulnerabilities

While the agent’s caching mechanism improves performance, it also introduces another layer of risk. Cached secrets are stored in-memory, which means that if an attacker gains access to a machine running the agent, they could extract secrets directly from memory without even interacting with the Secrets Manager, as long as they have access to the SSRF token. Without the token, the agent will not respond with a secret value. If the bad actor has root access to the computer, then all bets are off - this could lead to undetected exfiltration of secrets in the event of a breach.

5. Limited Visibility for Incident Response

The AWS Secrets Manager Agent can obscure certain logging details, which complicates incident response (IR) efforts. While the default logging behavior lacks sufficient detail, some visibility can be improved through additional configuration. However, two main issues arise: First, customers need to manually create a mechanism for streaming agent logs to a centralized location, unlike the AWS SDK, which automatically logs to CloudTrail by default. Second, during an incident, comparing logs from two different locations (the agent and CloudTrail) adds complexity and time overhead to the already critical and time-sensitive IR process. Additionally, because the agent doesn't facilitate two-way communication with the service, logs are spread across multiple places, making post-incident forensic analysis more fragmented and difficult. This makes it harder to adopt an 'assume breach' mindset and trace the exact source of a compromise efficiently.

Mitigation Strategies for Safe Deployment

Whenever possible, it's recommended to use the AWS SDKs for interacting with AWS Secrets Manager. The SDKs offer a more robust authentication and encryption layer compared to the Secrets Manager agent, while also facilitating seamless logging and event tracking in CloudTrail, which is invaluable for future incident response investigations.

If you decide to deploy the AWS Secrets Manager Agent, it's critical to adopt best practices to mitigate the risks discussed above:

  1. Evaluate trade-offs between security and availability before choosing the solution: If possible, use AWS SDKs, which offer more robust encryption and logging features. SDKs integrate seamlessly with AWS CloudTrail, providing detailed logs that are invaluable during IR.
  2. Secure Network Configurations: Ensure the agent operates within a dedicated network namespace, isolating it from other containers and reducing the risk of unauthorized access.
  3. Remove SSRF Tokens from Environment Variables: Prevent SSRF tokens from being exposed by removing them from process details. This adds an additional layer of security by reducing the risk of token theft via /proc/<PID>/environ access.
  4. Monitor CloudTrail Logs: Keep a close watch on CloudTrail logs. The agent’s unique User-Agent string (aws-secrets-manager-agent/) can be used to filter and monitor its activity for suspicious requests or behaviors.
  5. Use IAM Roles, Not User Credentials: Always configure the agent to assume the IAM role attached to the machine or container it's running on, avoiding the use of static IAM user credentials.
  6. Consider Adding TLS Support: For better security, consider using a modified version of the agent with TLS support for encrypted communication.

The Clutch Approach

At Clutch, we believe that securing and managing Non-Human Identities—including their interaction with secret management tools—is a cornerstone of product security. Our platform automatically monitors and alerts users to any suspicious activities involving the AWS Secrets Manager Agent, providing support to ensure that their environments are set up in the most secure way possible.

As part of our commitment to the security community, we’ve created a fork of the AWS Secrets Manager Agent to further enhance security by removing environment variables containing secrets from the process details. This significantly reduces the risk of token theft via environmental inspection.

Conclusion: A Double-Edged Sword

The AWS Secrets Manager Agent is a powerful tool that can reduce complexity for developers, but its security implications cannot be overlooked. Balancing ease of use with security best practices is crucial. Whether you choose to adopt this agent or stick with AWS SDKs, ensuring a secure configuration should be your top priority.

At Clutch, we're here to help navigate these complexities, offering universal solution to safeguard your NHIs and keep your enterprise landscape secure.

[1] Security is a shared responsibility between customers and AWS