Skip to main content
banner image
venafi logo

DevOps & Proliferation of Secrets [Keeping Data Safe]

DevOps & Proliferation of Secrets [Keeping Data Safe]

woman holding hand up to her hear to listen
October 2, 2019 | Guest Blogger: Helen Beal


‘Secrets’, in the context of cybersecurity and DevSecOps, describes the digital authentication credentials used in applications and services; anything you don’t want to share. The secrets you need to keep safe may include:

 

  • Usernames and passwords (also auto-generated passwords)
  • API tokens and other application keys/credentials (including within containers)
  • SSH Keys
  • Database and other system-to-system passwords
  • Private certificates for secure communication, transmitting and receiving of data (TLS, SSL, etc.)
  • Private encryption keys for systems like Pretty Good Privacy (PGP)
  • RSA and other one-time password devices
  • Keypad pin sequences
  • Symmetric encryption keys

 

 

These secrets need to stay secret because they are the doors to your data—if the wrong person gets hold of them, you have the potential to have an unwelcome intruder in your systems wreaking whatever havoc they can. Applications leak secrets under all sorts of circumstances: applications frequently log configurations, leaving them in log files or centralized logging systems. Often secrets will be captured in exception tracebacks or crash reports sent to external monitoring systems, or they will be leaked via debugging endpoints and diagnostic pages after hitting an error. Here are some examples of what can happen when secrets aren’t properly kept:

 

  • Uber hacked after leaving secrets on Github
  • Viacom left secrets publicly accessible on Puppet
  • Facebook stored passwords in plain text
  • Accenture left secrets exposed on Amazon S3
  • Pinterest built their own secrets management tool (Knox) in response to breaches

So, are you sure your secrets are safe? Can you answer these questions?

 

  • Where are all my secrets?
  • Who is accessing them and when?
  • Are they still there?
  • When did they get there?
  • Have they changed?
  • When were they removed?
  • How do I rotate all of these secrets to match security policies?


Learn how you can accelerate DevOps securely with HashiCorp and Venafi. Download the white paper.

You might now be thinking that it used to be easier to answer those questions. Historically the IP address was the unit of identity for security, and independent from any PKI identity. This was a workable approach when IPs were relatively static, and the rate of change was low. The challenge we are facing in the DevOps world is that our secrets are proliferating as a result of our desire to deploy changes more frequently. Our use of cloud and other modern technology practices (e.g. microservices, containers) makes our infrastructure more dynamic and ephemeral. We can’t use the IP address now because it’s no longer a static value from which we can base all our policies. It’s really hard for the technology teams to keep up even with the basic tasks of secrets generation, rotation, revocation, assignment, and sharing, particularly with all the other pressures on us to digitally transform, move from project to product, automate and evolve our cultures.
 

Causes of Proliferation

  • Machine Identity: The number of machines in organisations naturally proliferate through virtualization and containerization so these machines need their identities to be managed via secrets.
  • Cloud and Serverless: Teams developing cloud-native applications need to use secrets to mediate potentially thousands of services—which also vary by platform, so the challenge becomes ever more complex in a hybrid or multi-cloud environment.
  • Microservices: The modular services in a microservices environment are connected using HTTP or REST interfaces. This distributed model results in proliferation of interfaces and the communication between them generates several secrets management challenges. Additionally, each microservices module has its own database and credentials, thereby increasing the number of secrets to be managed.
  • DevOps: As infrastructure squads, IT operations teams and developers build Continuous Integration and Continuous Delivery (CI/CD) platforms, they must also secure these deployment pipelines.
  • AI/ML, IoT, Robotics and embedded devices: Burgeoning use of these new technologies compound the problem as every endpoint needs encryption and certificates too.
  • Blockchain: This too creates an increase in the number of private keys compared to a typical application driving the need for enterprise wallets.
     

Bad Behaviour: Secrets Sprawl

Secrets sprawl describes the surface area on which an organization’s secrets are stored along with the sheer volume of secrets as well as issues with secrets being duplicated and residing in multiple places—they have sprawled from their original and safe home. It is by no means uncommon for secrets to be stored in plain text in a variety of different places such as:
 

  • Files on individuals’ devices
  • Version control systems like GitHub*, Bitbucket or Azure DevOps
  • Configuration management platforms like Chef, Puppet, Ansible or Terraform
  • In CI/CD pipelines like Jenkins, TeamCity or Bamboo
  • On a sticky note (!)
     

*GitHub has a Secret Audit tool that you may find useful if this is your version control tool of choice. You could also take a look at TruffleHog.

 

Best Practices for Secrets Management in DevSecOps

Secrets should:
 

  • Never be stored in plain text
  • Never be transmitted over a network unencrypted
  • Have their value minimized
  • Be frequently and regularly rotated
  • Be as short-lived as possible
  • Be automatically revoked where possible
  • Have all lifecycle and access events recorded - fully auditable
  • Distributed by an authoritative delegator
  • Have limited operator access
  • Be versioned more easily than revealing plain text
  • Be mutually authenticated with related infrastructure components
  • Use role-based access control where possible
     

The best current way to ensure you and your teams are following these best practices is to use a Secrets Management tool for centralization. This kind of tool puts your secrets in a secret place, wraps that with encryption and access control, and provides fine-grained audit logging so you know who's doing what, when and only give access to people and applications that need it.
 

You should also encourage your teams to continuously reference the twelve-factor pattern to reduce application dependencies on credentials from external services. You can also consider using dynamic secrets: a dynamic secret is generated on demand and is unique to a client, instead of a static secret, which is defined ahead of time and shared. Dynamic secrets help reduce the blast damage of any leaked secrets or compromised systems because every authenticated entity will have a unique set of credentials.
 

Your secrets hierarchy design should account for secrets isolation per application, environment and a fail-proof revocation of secrets when required. To further strengthen the secrets structure, access policies and role-based mappings need to be built to support emergencies by making them version-controlled and automated.
 

Depending on your platform choices, you can take a look at Docker Secrets, AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, and CyberArk Conjur for secrets management.
 

It’s important that a secrets management tool has comprehensive and accessible APIs and integrations to other security tools such as machine identity protection tools like those offered by Venafi. The Venafi and HashiCorp Vault integration allows DevOps teams to obtain TLS keys and digital certificates using the Machine Identity Protection service operated by the organization’s security team: that’s DevSecOps in action.
 

Find out more about how Venafi and HashiCorp together make it faster and safer for DevOps teams to generate external certificates for their applications here and provide much needed policy controls and visibility for security teams.



 

Related posts

 

Like this blog? We think you will love this.
graphic of a man drawn in connected dots against a grey background. He is looking through a telescope up at the sky.
Featured Blog

Building Predictive Security into DevSecOps [Interview with Aaron Rinehart, Part 2]

Helen: How do you define DevSecOps?

Read More
Subscribe to our Weekly Blog Updates!

Join thousands of other security professionals

Get top blogs delivered to your inbox every week

See Popular Tags

You might also like

CIO Study: Certificate-Related Outages Continue to Plague Organizations
White Paper

CIO Study: Certificate-Related Outages Continue to Plague Organizations

Machine Identity Protection for Dummies
eBook

Machine Identity Protection for Dummies

About the author

Guest Blogger: Helen Beal
Guest Blogger: Helen Beal

Helen Beal is a DevOps guru. She currently serves as a Member of the DevOps World Advisory Board, the DevOps Institute Board of Regents, and is listed in PowerAdmin's 51 DevOps Influencers to Start Following Today.

Read Posts by Author
get-started-overlay close-overlay cross icon
get-started-overlay close-overlay cross icon
Venafi Risk assessment Form Image

Sign up for Venafi Cloud


Venafi Cloud manages and protects certificates



* Please fill in this field Please enter valid email address
* Please fill in this field Password must be
At least 8 characters long
At least one digit
At last one lowercase letter
At least one uppercase letter
At least one special character
(@%+^!#$?:,(){}[]~`-_)
* Please fill in this field
* Please fill in this field
* Please fill in this field
*

End User License Agreement needs to be viewed and accepted



Already have an account? Login Here

×
get-started-overlay close-overlay cross icon

How can we help you?

Thank you!

Venafi will reach out to you within 24 hours. If you need an immediate answer please use our chat to get a live person.

In the meantime, please explore more of our solutions

Explore Solutions

learn more

Email Us a Question

learn more

Chat With Us

learn more
Chat