Automate Your Certificate Lifecycle to Prevent DevOps Outages!
March 4, 2019|Guest Blogger: Kim Crawley
The prevention of certificate-related outages must be a top priority for DevOps. DevOps downtime can be very expensive and have terrible consequences. Production servers in particular must have as much uptime as possible. Damaging reliability and availability can harm an enterprise’s relationship with their clientele and become very costly. Recovering from a DevOps outage also takes time and labor hours out of the IT department, which could have been allocated more effectively had the outage been prevented.
Here are some frequently cited causes of DevOps outages:
Too much log data being simultaneously sent from network devices to a log server
Buggy scripts
Overwhelming a CPU by opening a folder within a file manager with way too many files
Improperly configured notification tools
Disk partitions being filled with huge core dumps
Networking problems like the one Microsoft reported pertaining to Azure: “The incident on Wednesday, 3 October 2018 started with a networking issue in the North Central US region. Since our authentication service, SPS, is in this region the issue impacted several of our scale units.”
Poor load balancing
Bad configuration rollouts
Operating failures of networking appliances, such as switches and firewalls
Network domino effects from database glitches
Improperly configured DNS changes
Broadcast storms from the incorrect usage of hubs and switches
Poor package management in production nodes
But there’s one type of failure that’s not on this list and it may be one of the most common causes of DevOps outages: poor TLS certificate issuance and management.
In a recent survey, 79% of respondents cited at least one TLS certificate-related DevOps outage per year. What’s even worse is that over a third of those respondents cited six certificate-related DevOps outages or more per year! So, what should DevOps teams do differently to prevent these outages?
The number of TLS certificates that a web application and other types of DevOps environments need to issue has greatly increased in recent years. Web browsers and other types of network client applications demand the encryption of network traffic now more than ever... and that’s a wonderful thing. Plaintext data in traffic is vulnerable to cyber attacks! Another factor that’s greatly increasing the rate of TLS certificate issuance is the explosion in the number of consumer and industrial Internet of Things (IoT) devices. If we’re going to have more and more IoT devices out there, they all need proper TLS certificates—which serve as machine identities—for strong encryption.
But if DevOps is stuck having to manually issue certificates, that’s a huge problem. Human beings are notoriously error-prone. And the manual configuration and issuance of certificates grinds the speed and efficiency of DevOps down to a halt. But there’s a sure way of preventing bottlenecks in your DevOps workflows and preventing one of the single most common causes of outages in DevOps.
Traditional certificate generation practices a littered with issues and incompatible with modern IT.
Modern DevOps leverages infrastructure-as-code for the continuous delivery of software. Virtual machines and containers often have very short lifespans which can be measured in hours or even just a few minutes. Certificate issuance and deployment now require a pull model, rather than a push model. Certificate issuance and deployment must be integrated with organizationally-approved certificate sources since certificate issuance is needed in real-time to support continuous deployments.
According to the pull model for certificate issuance, calls can be sent to centralized APIs for new certificate requests. Or certificate requests can be deployed by entities throughout the DevOps toolchain. Requests can originate from calls sent from Kubernetes, Terraform, Chef, Docker, or other popular DevOps tools. DevOps teams can get certificates using their preferred tools by environment and infrastructure type and there’s no need to leave the toolchain. There are so many very usable and efficient possibilities.
Many DevOps teams use solutions like HashiCorp Vault to automate certificate issuance, but that approach only works well for internal-facing infrastructure. Externally-facing load balancers and web servers require publicly-trusted certificates and they cannot be automated that way.
Policy compliant externally-facing certificates are key to preventing DevOps outages and increasing security. When organizationally-approved certificate sources are integrated with automated certificate issuance, each and every deployment can have checks to verify that none of the applicable certificates are up for expiration.
Each time the continuous integration/continuous delivery (CI/CD) pipeline runs, expiration checks can be performed and when new certificates are necessary, simple API calls can be performed to request and install that certificate on load balancers, web servers, and controllers.
Venafi Vice President of Security Strategy Kevin Bocek says, “Failure to renew certificates before they expire, and improper configurations (as with the Microsoft Azure outage) can be costly. Service failures with applications that use HTTPS can result in a downtime of up to $1 million per hour for high-volume services. To avoid unnecessary outages, make sure you are able to discover where all application certificates are in use and replace those that are unreliable or expired.”
It’s easy to understand. Centralizing and automating certificate issuance enable DevOps teams, PKI, and security teams to acquire full visibility of certificate usage throughout their networks. Problems can be diagnosed more easily, certificates can be more effectively monitored, and most importantly... DevOps outages can be prevented!
So, make sure certificate issuance and management is automated and embedded into your DevOps environments, it’s that simple!
CIO Study: Certificate-Related Outages Continue to Plague Organizations
About the author
Guest Blogger: Kim Crawley
Kim Crawley writes about all areas of cybersecurity, with a particular interest in malware and social engineering. In addition to Venafi, she also contributes to Tripwire, AlienVault, and Cylance’s blogs. She has previously worked for Sophos and Infosecurity Magazine.
PLEASE READ CAREFULLY BEFORE CONTINUING WITH REGISTRATION AND/OR ACTIVATION OF THE VENAFI CLOUD SERVICE (“SERVICE”).
This is a legal agreement between the end user (“You”) and Venafi, Inc. ("Venafi" or “our”). BY ACCEPTING THIS AGREEMENT, EITHER BY CLICKING A BOX INDICATING YOUR ACCEPTANCE AND/OR ACTIVATING AND USING THE VENAFI CLOUD SERVICE FOR WHICH YOU HAVE REGISTERED, YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ENTERING INTO THIS AGREEMENT ON BEHALF OF A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT THAT YOU HAVE THE AUTHORITY TO BIND SUCH ENTITY AND ITS AFFILIATES TO THESE TERMS AND CONDITIONS, IN WHICH CASE THE TERMS "YOU" OR "YOUR" SHALL REFER TO SUCH ENTITY AND ITS AFFILIATES. IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT AGREE WITH THESE TERMS AND CONDITIONS, YOU MUST NOT ACCEPT THIS AGREEMENT AND MAY NOT USE THE SERVICE.
You shall not access the Service if You are Our competitor or if you are acting as a representative or agent of a competitor, except with Our prior written consent. In addition, You shall not access the Service for purposes of monitoring its availability, performance or functionality, or for any other benchmarking or competitive purposes, and you shall not perform security vulnerability assessments or penetration tests without the express written consent of Venafi.
This Agreement was last updated on April 12, 2017. It is effective between You and Venafi as of the date of Your accepting this Agreement.
The Venafi Cloud Service includes two separate services that are operated by Venafi as software as a service, each of which is separately licensed pursuant to the terms and conditions of this Agreement and each of which is considered a Service under this Agreement: the Venafi Cloud Risk Assessment Service or the Venafi Cloud for DevOps Service. Your right to use either Service is dependent on the Service for which You have registered with Venafi to use.
Definitions
“Your Data” means electronic data and information submitted by or for You to the Service or collected and processed by or for You using the Service.
License Grants and Restrictions
License Grant by Venafi to You. Venafi grants to You a limited, non-exclusive, non-transferable, non-assignable, limited license (the “License”) to access and use the Service during the applicable License Term set out in Section 2.2 below, in accordance with the instructions contained in the user documentation that accompanies the Service (“Documentation). Venafi hereby grants to You the right to use the Documentation solely in connection with the exercise of Your rights under this Agreement. Other than as explicitly set forth in this Agreement, no right to use, copy, display, or print the Documentation, in whole or in part, is granted. This license grant is limited to internal use by You. This License is conditioned upon Your compliance with all of Your obligations under this Agreement. Except for the express licenses granted in this Section, no other rights or licenses are granted by Venafi, expressly, by implication, by way of estoppel or otherwise. The Service and Documentation are licensed to Licensee and are not sold. Rights not granted in this Agreement are reserved by Venafi.
License Term.
Venafi Cloud Risk Assessment Service. If you have registered to access and use the Venafi Cloud Risk Assessment Service, Your right to use the Venafi Cloud Risk Assessment Service is limited to ninety (90) days from the date You first register for the Service, unless otherwise extended on Your agreement with Venafi.
Venafi Cloud for DevOps Service. If you have registered to access and use the Venafi Cloud for DevOps Service, Your right to use the Venafi Cloud for DevOps Service shall extend indefinitely and may be terminated by either You or Venafi at any time for any reason.
Restrictions on Use. The grant of rights stated in Sections 2.1 and 2.2, above, is subject to the following restrictions and limitations:
If You have registered to access and use the Venafi Cloud for DevOps Service, You must use SSL/TLS certificates issued to you at no charge through the Service for development and testing purposes only, and You are strictly prohibited from using such SSL/TLS certificates in a production environment or in any production capacity. If you are registered with a public Certification Authority (“CA”) supported by the Service and have valid credentials issued by such CA with which you can subscribe to such CA’s SSL/TLS certificates on a fee bearing basis for use in production environments, You may request such certificates through the applicable interface present in the Service by using such credentials. In such instance, the fee bearing certificate(s) will be issued to You by the CA and any access to or use of such certificates by You will be subject to the terms and conditions set out by the CA. No fees will be paid to or processed by Venafi in this case. The use of DigiCert issued certificates shall be subject to the Certificate Services Agreement published by DigiCert at https://www.digicert.com/docs/agreements/Certificate-Services-Agreement.pdf, which terms are hereby incorporated by reference.
You shall not use (or cause to be used) the Service for the benefit of any third party, including without limitation by rental, in the operation of an Applications Service Provider (ASP) service offering or as a service bureau, or any similar means.
You shall not distribute access to the Service, in whole or in any part, to any third party or parties. You shall not permit sublicensing, leasing, or other transfer of the Service.
You shall not (a) interfere with or disrupt the integrity or performance of the Service or third-party data contained therein, (b) attempt to gain unauthorized access to the Service or its related systems or networks, (c) permit direct or indirect access to or use of the Service in a way that circumvents a contractual usage limit, or (d) access the Service in order to build a competitive product or service.
License Grant by You. You grant to Venafi and its affiliates, as applicable, a worldwide, limited-term license to host, copy, transmit and display Your Data as necessary for Venafi to provide the Service in accordance with this Agreement. Subject to the limited licenses granted herein, Venafi acquires no right, title or interest from You or any of Your suppliers or licensors under this Agreement in or to Your Data.
Ownership
Venafi Materials. Venafi and/or its suppliers have and shall retain ownership of all right, title and interest in and to the Service and the Documentation and all intellectual property rights embodied in the Service and Documentation, including without limitation any patents, copyrights, trademarks and trade secrets in the Service and any modifications and/or derivatives thereof, whether or not made at Licensee’s request, and all know-how, concepts, methods, programming tools, inventions, and computer source code developed by Venafi (collectively, “Venafi Materials”).
Limited Feedback License. You hereby grant to Venafi, at no charge, a non-exclusive, royalty-free, worldwide, perpetual, irrevocable license under Your intellectual property rights in and to suggestions, comments and other forms of feedback (“Feedback”) regarding the Service provided by or on behalf of You to Venafi, including Feedback regarding features, usability and use, and bug reports, to reproduce, perform, display, create derivative works of the Feedback and distribute such Feedback and/or derivative works in the Service. Feedback is provided “as is” without warranty of any kind and shall not include any of Your confidential information.
Disclaimer of Warranties
EXCEPT AS EXPRESSLY SET FORTH IN THIS SECTION 4, THE SERVICE AND DOCUMENTATION ARE PROVIDED “AS-IS,” WITH “ALL FAULTS” AND “AS AVAILABLE,” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, ACCURACY, RELIABILITY, OR NONINFRINGEMENT WHETHER ARISING FROM COURSE OF DEALING, USAGE, TRADE PRACTICE OR ANY OTHER MANNER. VENAFI IS NOT OBLIGATED TO PROVIDE ANY UPDATES, UPGRADES OR TECHNICAL SUPPORT FOR THE SERVICE. VENAFI DISCLAIMS ALL LIABILITY AND INDEMNIFICATION OBLIGATIONS FOR ANY HARM OR DAMAGES CAUSED BY ANY THIRD-PARTY HOSTING PROVIDERS. In no event does Venafi warrant that the Service is error free or that You will be able to operate the Service without problems or interruptions. Some jurisdictions do not allow the exclusion of implied warranties and to the extent that is the case the above exclusion may not apply.
Limitation of Liability
IN NO EVENT WILL VENAFI OR ITS SUPPLIERS BE LIABLE FOR ANY LOST REVENUE, PROFIT, OR DATA, OR FOR DIRECT, SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL, OR PUNITIVE DAMAGES HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY ARISING OUT OF THE USE OF OR INABILITY TO USE THE SERVICE EVEN IF VENAFI OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Some jurisdictions do not allow the limitation or exclusion of liability for incidental or consequential damages and to the extent that is the case the above limitation or exclusion may not apply to You.
Term and Termination
This License is effective until terminated as set forth herein or the License Term expires and is not otherwise renewed by the parties. Venafi may terminate this Agreement and/or the License at any time with or without written notice to You if You fail to comply with any term or condition of this Agreement or if Venafi ceases to make the Service available to end users. You may terminate this Agreement at any time on written notice to Venafi. Upon any termination or expiration of this Agreement or the License, You agree to cease all use of the Service if the License is not otherwise renewed or reinstated. Upon termination, Venafi may also enforce any rights provided by law. The provisions of this Agreement that protect the proprietary rights of Venafi will continue in force after termination.
Compliance With Laws
Violation of Laws. You shall not knowingly take any action or omit to take any action where the reasonably predictable result would be to cause Venafi to violate any applicable law, rule, regulation or policy and, to the extent not inconsistent therewith, any other applicable law, rule, regulation and policy.
Governing Law
This Agreement shall be governed by, and any arbitration hereunder shall apply, the laws of the State of Utah, excluding (a) its conflicts of laws principles; (b) the United Nations Convention on Contracts for the International Sale of Goods; (c) the 1974 Convention on the Limitation Period in the International Sale of Goods; and (d) the Protocol amending the 1974 Convention, done at Vienna April 11, 1980.
General
This Agreement is binding on You as well as Your employees, employers, contractors and agents, and on any permitted successors and assignees. Except if otherwise superseded in writing by a separately executed agreement, this Agreement is the entire agreement between You and Venafi with regard to the License granted hereunder, and You agree that Venafi will not have any liability for any statement or representation made by it, its agents or anyone else (whether innocently or negligently) upon which You relied in entering into this Agreement, unless such statement or representation was made fraudulently. This Agreement supersedes any other understandings or agreements, including, but not limited to, advertising, with respect to the Service. If any provision of this Agreement is deemed invalid or unenforceable by any country or government agency having jurisdiction, that particular provision will be deemed modified to the extent necessary to make the provision valid and enforceable and the remaining provisions will remain in full force and effect. Should such modification be impractical or denied, You and Venafi shall thereafter each have the right to terminate this Agreement on immediate notice.
Survival. The parties agree that the rights and obligations set forth in the above-referenced Section 1 (Definitions), 3 (Ownership), 4 (Disclaimer of Warranties), 5 (Limitation of Liability), 6 (Term and Termination), 7 (Compliance with Laws), 8 (Governing Law), and 9 (General) shall survive the termination of this Agreement for any reason and enforcement thereof shall not be subject to any conditions precedent.
Assignment. This Agreement shall be binding upon and inure to the benefit of the parties’ respective successors and permitted assigns. You shall not assign this Agreement or any of Your rights or obligations hereunder without the prior written consent of Venafi and any such attempted assignment shall be void.
This site uses cookies to offer you a better experience. If you do not want us to use cookies, please update your browser settings accordingly. Find out more on how we use cookies.