The Semantic Layer War for Security Data: The Compute Economics Behind Google SecOps Search Specifications

A Document That Teaches You How to Save Money Exposes a Structural Change Across the Entire Industry

Google Security Operations (formerly Chronicle) has published a document on UDM search best practices. It reads like typical vendor technical documentation: it discusses fields, operators, performance tuning, and troubleshooting.

But put back into context, this document is actually saying something bigger: on modern security operations platforms, search is no longer a free action.

The document opens with a warning—if queries are constructed poorly, search will consume substantial compute resources, and performance will also vary with the scale and complexity of the data in the instance. It then, almost in the tone of a cost consultant, lists item by item “high-performance fields,” “excluded fields,” and “how to narrow the time range.”

This is not grammar instruction. This is compute economics. When the official documentation of a security product needs to spend a great deal of space teaching users how to spend less money, compute less, and scan less data, it shows that the core contradiction in security operations has shifted from “can we see the attack?” to “can we afford to see it?”

Field Allowlist: Schema Is Infrastructure Power

The most informative part of the document is that it provides, for each type of query scenario, an explicit list of “high-performance UDM fields.” Metadata, principal, source, target, network, security result—six major categories of fields are listed one by one, with an explicit statement that these fields are “optimized for fast retrieval.”

This reflects a mature but often overlooked fact: a unified data model is not just a data specification; it is the boundary definition of query capability.

In the raw log world, any field can theoretically be searched, at the cost of scanning the full table. In the UDM world, some fields are wired into indexes and retrieval paths, while others live in the raw text and are parsed only when necessary. This layering is not a technical detail, but a bet by product designers on the future of security investigation—they believe what investigators truly use frequently are hostnames, IPs, file hashes, process IDs, user identifiers, email addresses, HTTP methods, and security rule names.

This list itself is a profile of “security investigation behavior”: the atomic unit of modern threat hunting is the correlatable entity identifier, not free text.

For competitors, this list is even more of a wall. Whoever defines the fields defines how detection rules are written, the analyst’s mental model, and the cost of migrating away from this platform. Schema has never been neutral; it is a concrete form of infrastructure power.

Deliberately Excluded Fields: The Exposed Corners of Compute EconomicsThere is a rare moment of candor in the documentation: certain fields are deliberately excluded from search filters, including metadata.id, metadata.product_log_id, and all *.timestamp fields. The reason is stated plainly—the unique values of these fields create high cardinality, which introduces statistical variance and hurts search performance.

This sentence is worth reading over and over.

In the field of observability and security data, high cardinality has always been the most stubborn adversary in the cost structure. A field containing millions or even billions of unique values is almost impossible to index efficiently. Vendors have only two choices when faced with it: either tough it out and pass the cost on to users, or keep it out of the fast retrieval layer and make users pay a higher price when they need it.

Google chose the latter, and wrote it into public documentation. This means it acknowledged one thing: in security data platforms, not all data is created equal.

More subtly, timestamps are placed in this category. Time should be the primary dimension of security investigations, but precisely because time can take nearly infinite values, it cannot serve as a cost-effective exact-match condition. The alternative given in the documentation is to compare using Unix epoch seconds, or to call YARA-L's time functions for date conversion—in other words, translating "human-readable time" into "machine-indexable integers," with users bearing that translation cost.

This is a typical engineering compromise, and also a typical business model choice.

Two Clocks: The Split Between Event Time and Ingestion Time

Another easily overlooked but far-reaching detail in the documentation is the explanation about "newly ingested data with older timestamps": the search time range is based on parsed event timestamps, not the ingestion timestamp of the raw logs.

It even gives a workaround—if you want to search newly ingested logs within old event timestamps, you must use the All time option and instead query metadata.ingested_timestamp.

Behind this is the replay, in a security context, of an old problem in stream processing: the separation of event time and processing time. In distributed systems, late logs, clock drift, proxy caching, and delayed uploads from offline devices all decouple data arrival time from event occurrence time.

In security investigations, however, this technical detail directly determines whether conclusions are right or wrong. In reconstructing the timeline of an intrusion, if analysts assume by default that the search window is divided by ingestion time, they may completely miss logs that attackers planted weeks ago and that were only uploaded today.A deeper issue is this: security platforms split the concept of “time” into two fields, which in effect hands part of the interpretive authority over temporal semantics back to users. This is both flexibility and a transfer of responsibility. When the platform no longer decides for you “which time matters,” analysts must answer a question that should belong to data engineering.

The Split Between Entities and Events: Context Data Is Treated Separately

The documentation clearly states: if you use metadata.log_type = "..." to search log types such as AZURE_AD_CONTEXT and WORKSPACE_USERS that are entities or context, it will return empty results—because UDM search queries only UDM event records. To access this data, you must instead use graph syntax, through paths such as graph.metadata.event_metadata.log_type or graph.entity.user.email_addresses.

This detail reveals a bifurcation in security data architecture: events and entities are being stored and queried separately.

Events are “what happened”; entities are “who is who.” For the past decade, SIEM has mainly handled the former. But modern security operations increasingly rely on the latter—identity graphs, asset context, and user profiles. Attackers rarely expose themselves with just one event; their traces are scattered across large numbers of events, and only by linking events together through entities do anomalies surface.

Querying entity data with a completely different syntax is an architectural signal: entity data is not an appendage of events, but another data structure that requires independent modeling and independent optimization. This also explains why graph querying is becoming a standard capability for security platforms, not just a tool exclusive to threat intelligence.

For the startup ecosystem, there is a clear window of opportunity here: entity resolution, identity normalization, and cross-source entity alignment are becoming a layer of infrastructure independent of SIEM.

Repeated Fields and the Silent Trap of any

The documentation contains a passage about “repeated fields”; its tone is matter-of-fact, but the consequences are not trivial. Fields such as principal.ip and target.file.md5 can store multiple values in a single event. By default, they are evaluated using the any operator: as long as any one value in the field satisfies the condition, the entire predicate is true.The example given in the documentation is highly representative: search for principal.ip != "1.2.3.4", and if an event contains both 1.2.3.4 and 5.6.7.8, that event will still be matched. Because 5.6.7.8 satisfies the "not equal" condition.

For human analysts, this is almost counterintuitive. For engineers building automated detection rules, this is a trap that can directly lead to false positives and false negatives.

It reveals a more general problem: the semantics of security query languages is becoming a discipline that requires dedicated study. When the behavior of logical operators depends on the internal storage structure of the data model, detection engineering is no longer just "writing correct logic," but "understanding how the underlying data structures evaluate."

This also explains why the documentation specifically mentions that there is an upper limit on logical operators inside parentheses (169), why regular expressions cannot be used on enumeration fields, and why the behavior of the RE2 engine needs to be documented separately. The complexity of security query languages is approaching that of a small programming language—and every new language gives rise to a new specialty.

The Prerequisites for AI Investigation Agents Are Written in This Document

If you read this document as laying the groundwork for the future, its most interesting aspect is that it inadvertently defines the conditions that must first be met for an AI security agent to function.

Condition one: query costs must be predictable. If every exploratory query by an autonomously operating investigation agent could trigger uncontrollable compute consumption, no one would dare let it act freely. Field whitelists, time-window constraints, and excluded high-cardinality fields are essentially all about making the cost model budgetable.

Condition two: semantics must be deterministic. The default behavior of the any operator, element-by-element evaluation of repeated fields, and the distinction between event time and ingestion time—these are all rules that the agent must grasp precisely and cannot be vague about. Machines cannot bypass semantic traps through intuition.

Condition three: interfaces must be structured. The documentation recommends using reference lists instead of large-scale value matching, Unix epoch instead of human-readable time strings, and explicit field paths instead of free-text search—all of these are transforming "investigation" from a vague human activity into a set of deterministic steps that can be invoked by programs.

In other words, the concept of an agentic SOC is not that first there is an agent, and then it adapts to the platform. The real order is exactly the opposite: the platform first organizes the data model, query semantics, and cost boundaries into a machine-usable form, and only then does the agent have somewhere to land. This document is part of that preparation work, even though it does not say so itself.

Schema Standardization: A Quiet but Critical Competition

Zoom out: the battlefield this document occupies is the standardization competition for security data schemas.

On one side are vendor-proprietary schemas, such as Google's UDM. Their advantage is deep optimization and tight integration with their own search and detection engines; the cost is migration expense and ecosystem lock-in. On the other side is the industry-driven push toward open schema directions, with the goal of giving logs from different sources consistent field semantics and lowering the barrier to cross-platform collaboration.

This competition will not play out before the public through launch events or marketing battles. It shows up in drier places: field naming, the list of searchable fields, how timestamps are represented, whether entities and events are separated, and the choice of regex engine.

But these details determine the next generation's reality: whether detection rules can be reused across platforms, whether threat intelligence can be exchanged between organizations, whether AI agents can understand data from different vendors, and how much negotiating room security teams still have before they are locked in.

Conclusion: Security operations is shifting from a storage business to a semantic-layer business

For the past two decades, the core narrative of SIEM was "centralize the logs." Storage was a cost, and search was a feature.

Now the relationship has reversed. Storage is relatively cheap, and making data queryable, correlatable, and machine-understandable is the truly expensive and truly valuable part.

The value of this Google SecOps document is not in what syntax it teaches, but in how honestly it exposes the new rules of the game: fields have tiers, time is ambiguous, queries have a price, and semantics have boundaries.

Whoever can define these boundaries is not merely selling a security platform; they are defining the grammar of security investigation itself. That is more worthy of attention than any product launch.

---

Source: Search best practices | Google Security Operations