← All guides
How-to3 min read

Extracting very large datasets

Period filters, file splitting and running batches side by side.

How long will it take?

It depends on how many years you're extracting, the speed of your server and where you run the tool from. Master files and open items are quick, usually minutes. Transaction history takes the longest. If you've used Exchequer for ten years or more, a full extract can run overnight or over a weekend.

Four ways to make it easier

  1. Extract only what you need. Use the period filter (from and to, as period and year) to skip years you don't need.
  2. Split the output. Set the maximum records per file. 1,000 works well and keeps each file easy to open. The tool numbers them for you, such as Customers_0000001, Customers_0000002.
  3. Run batches side by side. You can run the tool more than once at the same time, for example one copy per range of years.
  4. Start the big run at the end of the day. Let it work overnight and check the results in the morning.

If you see "Out of Memory"

On very large datasets, Exchequer itself can run out of memory part-way through a long extract. It's caused by how Exchequer manages memory, not by the tool. The fix is simple: split the extract into smaller period ranges.

Run 1Years 1 to 4
Run 2Years 5 to 8
Run 3Years 9 to 12

Start all three at once and they'll finish sooner than one long run.

After the run

Check the task list on screen. Each task shows how many records were exported, plus any that failed and why. Some warnings are minor, such as "no jobs found". If anything needs a closer look, email us the details.

Get a feel for timings first The free demo runs every step on your own Exchequer.
Download the free demo

Hit a problem mid-extract?

Send us a screenshot of the task list and we'll help you sort it.

Talk to us