Indexify - By Tensorlake

Effortless Ingestion and Extraction At Any Scale For LLMs

Indexify is a an open source data framework featuring a real-time extraction engine and pre-built extraction adapters. Out of the box reliable extraction for every form of unstructured data(documents, presentation, videos and audio).

Get Started Download

Start Indexify Server & ExtractorsCreate Extraction GraphIngest Documents, Videos & TextRetrieve

Use any of the pre-built extractors or your custom extractors to transform or extract data from unstructured sources.

from indexify import IndexifyClient, ExtractionGraph
client = IndexifyClient()

extraction_graph_spec = """
name: 'sec10k'
extraction_policies:
  - extractor: 'tensorlake/chunk-extractor'
    name: 'chunks'
    input_params:
    text_splitter: recursive
  - extractor: 'tensorlake/minilm-l6'
    name: 'embedding'
    content_source: 'chunks'
"""
extraction_graph = ExtractionGraph.from_yaml(extraction_graph_spec)
client.create_extraction_graph(extraction_graph)

Keep your LLM powered
application ahead of constantly
changing data

To keep responses accurate, LLMs need access to up to date data. Indexify extracts continuously in near real-time (< 5ms) to ensure the data your LLM application depends on is current, without you needing to think about CRON jobs or reactivity.

Extract from video,
audio, and PDFs

Indexify is multi-modal and comes with pre-built extractors for unstructured data, complete with state of the art embedding and chunking. You can create your own custom extractors using the Indexify SDK, too.

Query using SQL and
semantic search

Just because your data is unstructured doesn't mean it needs to be difficult to retrieve. Indexify supports querying images, videos, and PDFs with semantic search and even SQL, so your LLMs can get the most accurate, up to date data for every response.

From prototype to
production

Indexify runs just as smoothly on your laptop as it does across 1000s of autoscaling nodes.

Start prototyping with Indexify’s local runtime and when you are ready for production, take advantage of our pre-configured deployment templates for K8s (or VMs) or even bare metal. Everything is observable out of the box, whether its ingestion speed, extraction load or retrieval latency.

Multi-Cloud for Better
Economics and Availability

Cost efficiency for LLMs today is about using the right hardware for the right parts of your stack at the best price points. Deploy Indexify across multiple clouds for maximum flexibility.