How It Works

Like many similar projects, SearchReplica abuses Logical Replication protocol, decoding all the transactions commited to Write-Ahead-Log and forwarding them into Elasticsearch/OpenSearch in a form a JSON documents

Following is the simplified workflow.

1. Fetch and parse all comments related to puslished tables using monstrous query.
2. Check if key fields are awailable in WAL (see: Upsert Mode)
3. Discover User-Defined types, like enums or composite types.

1. (re)Create Replication Slot if needed.
2. Fetch all existing data via COPY TO command. And process them 1 by 1.
3. Push to Elasticsearch in batches

After initial cold-start reindexing

1. Subscribe
2. Get Messages in a loop and process them 1 by 1.
3. Periodically push to Elasticsearch in batches.
4. After successfull push, commit LSN of last pushed document back to postgres

This approach provides you with near-to-realtime consistent data without additioan efforts.