Embed

Embed is the query-side half of schema-attribute embedding: a rank_by function that turns query text into a vector server-side, so the caller never runs its own embedding step. It matches Turbopuffer’s native-embeddings wire (private beta at turbopuffer.com/docs/embedding) verbatim, so a request written against tpuf’s Embed examples works against Layer unchanged. The gateway intercepts it on every backend, so behavior is identical on hev search and Turbopuffer namespaces.

Using Embed

// target column's declared model is implied
"rank_by": ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm"]]

// tpuf shorthand: the source attribute name also resolves
"rank_by": ["text", "ANN", ["Embed", "chest pain radiating to left arm"]]

// explicit model override, tpuf extended form
["Embed", "chest pain radiating to left arm", { "model": "Snowflake/snowflake-arctic-embed-m-v1.5" }]

response = await client.query_namespace("clinical-notes", {
    "rank_by": ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm"]],
    "top_k": 10,
})
print(response.rows, response.usage.embedding_tokens)

response, err := client.QueryNamespace(ctx, "clinical-notes", &hevlayer.QueryRequest{
    RankBy: []any{"embed_text", "ANN", []any{"Embed", "chest pain radiating to left arm"}},
    TopK:   10,
})

const response = await client.queryNamespace("clinical-notes", {
  rank_by: ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm"]],
  top_k: 10,
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/clinical-notes/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm"]],
    "top_k": 10
  }'

The gateway resolves Embed to a concrete vector before the backend sees the query — the store, hev search or Turbopuffer, always receives a plain ANN query. If the target column declares instructions.query (asymmetric models), that prefix is applied before embedding. Query embeddings are cached by model + revision + instruction + text with a short TTL, so repeated and paginated queries don’t re-tokenize.

Embed and Auto routing

Query routing’s Auto spelling defers to the caller when a chosen route needs a vector the request didn’t supply (routing.executed: false). Passing Embed inline as the vector source retires that round trip for any column with a declared embed: profile — the semantic and fused legs resolve and execute in one request. Deferral remains the answer only when the target column has no embedding profile; query routing covers that path.

Model resolution

A model name in Embed’s {model} option resolves against the models declared on the namespace’s schema (embed.model on the attribute, Namespace CRD) — not against a fixed provider menu. Omitting {model} resolves to the target column’s own declared model; passing a model id that isn’t declared on any attribute of the namespace is a 422 naming the declared models. There is no cross-namespace model catalog: the same model id can point at different pinned revisions on different namespaces.

Token accounting

Write and query responses echo tokens embedded, mirroring tpuf’s response echo:

{
  "rows": [ /* ... */ ],
  "usage": {
    "embedding_tokens": 8
  }
}

The exact field name and placement track the tpuf beta wire — compatibility here means matching their shape, not inventing an adjacent one, and the beta’s shape is not yet finalized upstream. Treat usage.embedding_tokens as the settled-by-this-doc name pending that confirmation.

There is no hevlayer_embed_tokens* Prometheus metric or cost.rs rate line in the gateway today — the write and query paths do not compute vectors yet (embed: and Embed are the shape this doc settles). The closest existing precedent is hevlayer_hybrid_text_tokens (a per-query token histogram for hybrid-text fusion) and hevlayer_agent_tokens_total (a counter of model tokens by agent/turn/token_type) — neither is wired into cost.rs, so Embed/embed: land in the same gap agent tokens are already in.

Embedding cost

Embedding cost should appear in the gateway’s own cost API, the same way Turbopuffer storage, write, and query cost already do (apps/layer-gateway/src/cost.rs’s TURBOPUFFER_RATE_CARD and hevlayer_cost_usd_per_hour), rather than leak to a separate provider invoice — the RFC’s “price:performance is ours to measure and own” line. This doc settles the shape an operator should expect to land on; none of it exists in cost.rs yet:

• hevlayer_embed_tokens_total — counter, labels namespace, store_kind, model, serving (the serving policy in effect: native | autoscaler | blended). Total tokens embedded, write and query paths combined.
• hevlayer_embed_compute_seconds_total — counter, same labels. For GPU-resident models where cost tracks wall-clock compute rather than tokens (image/CLIP embedding has no natural token unit).
• A rate card line per model class, alongside TURBOPUFFER_RATE_CARD in cost.rs — $/M tokens for CPU-tier text models, $/GPU-second for GPU-resident and CLIP models, priced from the hevlayer-inference pool’s instance cost rather than a per-vendor list.
• Composition with serving policy: a native-served model’s cost is the pool’s fixed steady-state instance cost, amortized — visible, but not additively metered per token the way autoscaler cost is. An autoscaler-served model’s cost is metered per token/compute-second against the scale-to-zero pool, plus the cold-start seconds paid on the first request after idle. blended reports both: the native-shaped amortized cost for the shift window’s traffic, then the autoscaler numbers once traffic has moved — an operator reading the two series together sees the shift happen, rather than one blended-and-opaque number.

Open question carried back to the RFC: whether embedding_tokens in the response usage object and hevlayer_embed_tokens_total in Prometheus are required to agree exactly per-request, or whether client-side batching makes that guarantee too strong to promise. See the open-questions note appended to docs/rfcs/embed-wire.md.