Incident report on March 13, 2024

Edit March 18, 2024 10:20 AM

We've detected from our logs that 91 GitHub tokens were compromised. The users have been notified, and we're working with GitHub to identify whether the tokens were used to access private repositories.

Summary (TL;DR)

On March 1st, we received an email raising concerns about the security of our endpoints. This in turn prompted us to rake through our logs and we discovered unusual requests to our servers originating from an unrecognized device.

Alarmingly, we noticed that some of these requests targeted sensitive API endpoints and were successful in their attempts. This unusual activity indicated that the actor behind these requests had possession of our private admin access tokens, granting them unauthorized access to our endpoints.

We received confirmation that GitHub tokens stored within our databases were used to access a customer's repository. While we do not have evidence of any other such instances, we cannot confirm that no other such instances occurred.

We took immediate action by revoking all GitHub token access, rotating our admin access tokens, and implementing stringent security measures to all of our APIs to mitigate any further unauthorized access. Additionally, we have patched the vulnerability that led to the exposure of our admin access tokens.

We have also since partnered with third-party cybersecurity vendors to conduct an extensive investigation, and have implemented other security measures to ensure that this type of unauthorized access cannot occur again. For the security of our users, we decided to implement those security measures before making this public announcement.

Timeline

All timestamps referenced are in Pacific Daylight Time (PDT).

Friday, March 1 4:55PM - Received an email raising concerns about the security of our endpoints/potential leaking of our token.
Friday, March 1 6:41PM - Discovered logs of an unrecognized device accessing API endpoints.
Friday, March 1 6:51PM - Revoked all existing GitHub user access tokens.
Friday, March 1 6:51PM-11PM
- Rotated our internal access tokens.
- Enhanced security protocols around endpoint authorization to prevent unauthorized access.
- Got in contact with a couple bug bounties.
Saturday, March 2nd and 3rd - Continued to stay in close contact with a bug bounty reporter, patched the vulnerability that resulted in the leak of our access token and revoked and rotated all tokens again.

How this affects you

No further action is required on your part to continue using our product safely.

Our team has addressed the vulnerability and taken steps to secure our systems against similar incidents in the future.

Actions and Remediations

In our response to protect our users and our systems, these are the measure that we have already taken:

Revoked all existing GitHub user access tokens.
Rotated of our internal access tokens.
Patched the vulnerability that resulted in the leak of our access token.
Enhanced security protocols around endpoint authorization to prevent unauthorized access.
Received a penetration test.

These are ongoing preventing measures that we are currently taking:

Collaborating with leading cybersecurity firms, including Oneleet, and our other partners, to conduct a thorough investigation and fortify our defenses against potential future attacks.
Enhancing the monitoring and alerting systems for our API endpoints to detect and respond to unusual activities swiftly.
Developing a comprehensive security policy and establishing a public page dedicated to outlining our security measures and protocols.
Launching a bounty program to facilitate the reporting of security vulnerabilities from ethical hackers.
Re-auditing our SOC 2 certification for 2024.

Conclusion

We deeply regret the inconvenience and concern this incident may have caused. Our dedication to transparency, security, and the trust you place in us remains unwavering.

Your security and trust are the foundations upon which Mintlify is built. We are dedicated to ensuring the continued safety and security of your content and information.

Should you have any concerns or questions, please do not hesitate to contact us at [email protected].

Sincerely,
The Mintlify Team