← Vectorize / vectorize / reference

Metadata filtering

In addition to providing an input vector to your query, you can also filter by vector metadata associated with every vector. Query results will only include vectors that match the filter criteria, meaning that filter is applied first, and the topK results are taken from the filtered set.

By using metadata filtering to limit the scope of a query, you can filter by specific customer IDs, tenant, product category or any other metadata you associate with your vectors.

Vectorize supports namespace filtering by default, but to filter on another metadata property of your vectors, you'll need to create a metadata index. You can create up to 10 metadata indexes per Vectorize index.

Metadata indexes for properties of type string, number and boolean are supported. Please refer to Create metadata indexes for details.

You can store up to 10KiB of metadata per vector. See Vectorize Limits for a complete list of limits.

For metadata indexes of type number, the indexed number precision is that of float64.

For metadata indexes of type string, each vector indexes the first 64B of the string data truncated on UTF-8 character boundaries to the longest well-formed UTF-8 substring within that limit, so vectors are filterable on the first 64B of their value for each indexed property.

An optional filter property on query() method specifies metadata filters:

• filter must be non-empty object whose compact JSON representation must be less than 2048 bytes.
• filter object keys cannot be empty, contain " | . (dot is reserved for nesting), start with $, or be longer than 512 characters.
• For $eq and $ne, filter object non-nested values can be string, number, boolean, or null values.
• For $in and $nin, filter object values can be arrays of string, number, boolean, or null values.
• Upper-bound range queries (i.e. $lt and $lte) can be combined with lower-bound range queries (i.e. $gt and $gte) within the same filter. Other combinations are not allowed.
• For range queries (i.e. $lt, $lte, $gt, $gte), filter object non-nested values can be string or number values. Strings are ordered lexicographically.
• Range queries involving a large number of vectors (~10M and above) may experience reduced accuracy.

Both namespaces and metadata filtering narrow the vector search space for a query. Consider the following when evaluating both filter types:

• A namespace filter is applied before metadata filter(s).
• A vector can only be part of a single namespace with the documented limits. Vector metadata can contain multiple key-value pairs up to metadata per vector limits. Metadata values support different types (string, boolean, and others), therefore offering more flexibility.

{ "streaming_platform": "netflix" }

{ "someKey": { "$ne": "hbo" } }

{ "someKey": { "$in": ["hbo", "netflix"] } }

{ "someKey": { "$nin": ["hbo", "netflix"] } }

{ "timestamp": { "$gte": 1734242400, "$lt": 1734328800 } }

Range queries can implement prefix searching on string metadata fields. This is also like a starts_with filter.

For example, the following filter matches all values starting with "net":

{ "someKey": { "$gte": "net", "$lt": "neu" } }

{ "pandas.nice": 42, "someKey": { "$ne": "someValue" } }

{ "pandas.nice": 42 }

// looks for { "pandas": { "nice": 42 } }

With the following index definition:

npx wrangler vectorize create tutorial-index --dimensions=32 --metric=cosine

Create metadata indexes:

npx wrangler vectorize create-metadata-index tutorial-index --property-name=url --type=string

npx wrangler vectorize create-metadata-index tutorial-index --property-name=streaming_platform --type=string

Metadata can be added when inserting or upserting vectors.

const newMetadataVectors: Array<VectorizeVector> = [
	{
		id: "1",
		values: [32.4, 74.1, 3.2, ...],
		metadata: { url: "/products/sku/13913913", streaming_platform: "netflix" },
	},
	{
		id: "2",
		values: [15.1, 19.2, 15.8, ...],
		metadata: { url: "/products/sku/10148191", streaming_platform: "hbo" },
	},
	{
		id: "3",
		values: [0.16, 1.2, 3.8, ...],
		metadata: { url: "/products/sku/97913813", streaming_platform: "amazon" },
	},
	{
		id: "4",
		values: [75.1, 67.1, 29.9, ...],
		metadata: { url: "/products/sku/418313", streaming_platform: "netflix" },
	},
	{
		id: "5",
		values: [58.8, 6.7, 3.4, ...],
		metadata: { url: "/products/sku/55519183", streaming_platform: "hbo" },
	},
];

// Upsert vectors with added metadata, returning a count of the vectors upserted and their vector IDs
let upserted = await env.YOUR_INDEX.upsert(newMetadataVectors);

Use the query() method:

let queryVector: Array<number> = [54.8, 5.5, 3.1, ...];
let originalMatches = await env.YOUR_INDEX.query(queryVector, {
	topK: 3,
	returnValues: true,
	returnMetadata: 'all',
});

Results without metadata filtering:

{
	"count": 3,
	"matches": [
		{
			"id": "5",
			"score": 0.999909486,
			"values": [58.79999923706055, 6.699999809265137, 3.4000000953674316],
			"metadata": {
				"url": "/products/sku/55519183",
				"streaming_platform": "hbo"
			}
		},
		{
			"id": "4",
			"score": 0.789848214,
			"values": [75.0999984741211, 67.0999984741211, 29.899999618530273],
			"metadata": {
				"url": "/products/sku/418313",
				"streaming_platform": "netflix"
			}
		},
		{
			"id": "2",
			"score": 0.611976262,
			"values": [15.100000381469727, 19.200000762939453, 15.800000190734863],
			"metadata": {
				"url": "/products/sku/10148191",
				"streaming_platform": "hbo"
			}
		}
	]
}

The same query() method with a filter property supports metadata filtering.

let queryVector: Array<number> = [54.8, 5.5, 3.1, ...];
let metadataMatches = await env.YOUR_INDEX.query(queryVector, {
	topK: 3,
	filter: { streaming_platform: "netflix" },
	returnValues: true,
	returnMetadata: 'all',
});

Results with metadata filtering:

{
	"count": 2,
	"matches": [
		{
			"id": "4",
			"score": 0.789848214,
			"values": [75.0999984741211, 67.0999984741211, 29.899999618530273],
			"metadata": {
				"url": "/products/sku/418313",
				"streaming_platform": "netflix"
			}
		},
		{
			"id": "1",
			"score": 0.491185264,
			"values": [32.400001525878906, 74.0999984741211, 3.200000047683716],
			"metadata": {
				"url": "/products/sku/13913913",
				"streaming_platform": "netflix"
			}
		}
	]
}

• As of now, metadata indexes need to be created for Vectorize indexes before vectors can be inserted to support metadata filtering.
• Only indexes created on or after 2023-12-06 support metadata filtering. Previously created indexes cannot be migrated to support metadata filtering.