DigitalOcean + Let's Encrypt

DigitalOcean’s commitment to simplifying cloud computing and providing an excellent experience for our customers is at the forefront of all we do. We’re dedicated to the success of startups and SMBs and continuously seek ways to evolve our products and partnerships so that we can enable teams to grow faster. Support of open source projects is a significant portion of that mission, and we’re proud to give back to the community by sponsoring projects that help developers spend more time doing what they love: building great software.

That’s why we use and sponsor Let’s Encrypt, a Certificate Authority built by the Internet Security Research Group (ISRG), providing free, automated certificates for everyone. Certificate Authorities play a crucial role in securing the Internet, and Let’s Encrypt changed the game by providing not only publicly trusted certificates for free but also a standardized API that enables automation, reducing the burden of certificate maintenance operations.

In this post, we’ll walk through where DigitalOcean uses Let’s Encrypt, how our integration works, and some of the enhancements we’ve made along the way.

DigitalOcean’s integration with Let’s Encrypt

Four years ago, we added Let’s Encrypt support to our Load Balancers product, which brought free, automatically renewed certificates for all our customers. Prior to that, customers’ only choice for certificates was to bring and manage their own certificates, which is operationally burdensome and can lead to downtime if they’re not monitoring the expiry of all their certificates. By using our Let’s Encrypt integration, customers no longer have to worry about rotating their certificates; just create the certificate once, configure the Load Balancer to use it, and DigitalOcean takes care of the rest.

Since then, our integration with Let’s Encrypt has expanded in the following ways:

• The Spaces product’s CDN feature can also use Let’s Encrypt certificates

• Support for wildcard domains

• Migrated to ACME v2 API

• Managed MongoDB now uses Let’s Encrypt certificates, enabling more seamless and secure connections to clusters

As of today, over two-thirds of all customer certificates use the Let’s Encrypt integration.

Architecture

So how does our integration with Let’s Encrypt actually work under the hood?

As shown in the architecture diagram, there are quite a few services involved. Let’s walk through the creation of a Let’s Encrypt certificate.

First, the customer requests the creation, either through the Cloud UI or the API (doctl, Terraform, etc.). After a few intermediate services that aren’t pictured here, the request lands at the Certificates and lets-encrypt-certs set of backend services.Next, the Certificates service validates the request to ensure the provided certificate is actually usable and not a duplicate of one the customer already has.

Then the request is forwarded to the lets-encrypt-certs service, which does the following:

• Stores some metadata in the MySQL database.

• Calls the Let’s Encrypt ACME v2 API, using the open source Go x/crypto/acme package to create and fetch the authorization order, passing along the customer’s provided list of domain names. The Let’s Encrypt ACME v2 API responds with a challenge token.

• Because we use the DNS-01 challenge type for Domain Validation, lets-encrypt-certs next calls the Domains service to create a TXT record on the customer’s _acme-challenge subdomain, with the challenge token used as the record value.

• Generates a new public/private key pair, sends the associated Certificate Signing Request to the ACME v2 API, and parses the resulting certificate from the API response.

• If any of the steps fail, we asynchronously retry them up to 3 hours later.

After that is complete, the Certificates service encrypts the sensitive fields, like the generated private key, before storing it and the PEM-encoded certificate in the MySQL database. Finally, the Certificates service sends an email to the customer to notify them that their certificate has successfully been provisioned.

Now that the certificate has successfully been created, the customer can reference it when configuring their Load Balancer or Spaces bucket.

Creating the certificate is just part of the problem, though. Arguably the most important aspect that we handle on behalf of our customers is automated certificate renewal. With Let’s Encrypt issuing certificates with 90-day lifespans by design, we need something in place to proactively renew certificates and distribute them to customer Load Balancers, Spaces buckets, and MongoDB clusters.

For that, we run a job called lets-encrypt-renewer every four hours that check the MySQL database for any certificates that are between 30 and 90 days old. This means we are renewing at most 60 days sooner than is absolutely necessary; however, because something can always fail, we provide plenty of time for retry attempts.

The job then largely follows the same steps listed above. Upon successful completion of the renewal, however, it additionally publishes a message to a Kafka topic. Backend services of the three consuming products—Load Balancers, Spaces, and Managed MongoDB—all listen for events on that topic and handle the subsequent distribution of the new certificate accordingly.

Migrating to ECDSA

Earlier this year, we migrated all DigitalOcean customer Let’s Encrypt certificates from RSA with 2048-bit keys to ECDSA using curve P-256. Computationally, ECDSA is significantly more efficient than RSA—that is, a server has to spend fewer CPU cycles to compute ECDSA signatures than RSA signatures. Additionally, P-256 actually provides a higher security level, equivalent to RSA-3072, and has become the industry standard for ECDSA certificates in the past few years.

Over 70% of the certificates attached to Load Balancers were using the Let’s Encrypt integration; this provided us with a big opportunity to improve our product for a lot of our customers. Because we absorb the computational cost of running Load Balancers, every bit of reduction in CPU usage means less overhead for us and better overall performance of our Load Balancers. This has the added benefit of freeing up CPU for regular droplets of customers in the rest of the fleet and reducing the time taken for TLS handshakes also generally yields faster page loads for end users.

Over the course of 30 days, we upgraded all existing customer Let’s Encrypt certificates to ECDSA during the regular renewal process. On our heaviest-usage Load Balancers, CPU usage came down by over 30% with this migration.

Managed MongoDB

With our Managed MongoDB product, customers connect to their database cluster over TLS. When we first released our managed Managed MongoDB product, we created certificates for each node, signed by a self-signed certificate authority, and provided to customers. This led to an inconvenient and confusing “out of the box” user experience: customers had to download and install this root certificate on all their clients before being able to connect to their DB.

In May, we replaced the self-signed certificates with Let’s Encrypt certificates instead. Because Let’s Encrypt’s root CA— ISRG Root X1—is already trusted by nearly every client out there, customer connections to their databases will just work—no extra configuration needed! And, since these certificates are ECDSA, database clusters can spend fewer CPU cycles on TLS handshakes and more time processing queries.

What’s next?

One of the biggest requests we receive from customers of our Certificates product is the ability to create Let’s Encrypt certificates without having to manage their domain with us. At the moment, this prerequisite exists so that we can create the TXT record for DNS-01 challenges on the appropriate sub-domains on behalf of the customer. We’re looking into potential solutions around this limitation, like leveraging CNAME or NS records or using HTTP challenges instead.

In the meantime, check out some of our Let’s Encrypt community tutorials! Or, if you need Let’s Encrypt certificates in your DOKS cluster, consider using our Marketplace app for the popular open-source project, cert-manager.

A great partnership

We want to give a big thank you to our partners at Let’s Encrypt for continuing to work with us on improving the certificate experience for everyone. As a nonprofit, 100% of their funding comes from charitable contributions—if you or your organization want to get more involved in supporting their mission, visit this website.

Interested in building the cloud at DigitalOcean? Check out our careers page for openings on our teams!