Large Areas: Size-Aware and Tiled Processing

Rwapor decides how to process a job from its size. A 5 km farm at 20 m fits in memory and runs without tiling; a 150 km irrigation scheme at 20 m, or several years of dekads, is streamed or split into tiles automatically. The results are the same whichever mode runs; only memory use and speed change.

See the plan before you run

wapor_plan_processing() estimates the memory a job needs and reports the mode it would use, with its reasons.

library(Rwapor)

# A small farm at 20 m, one year of dekads
wapor_plan_processing(aoi = c(35.95, 33.80, 36.00, 33.85), resolution = 20, n_layers = 36)

# A large scheme at 20 m, four variables
wapor_plan_processing(aoi = c(35.0, 33.0, 36.5, 34.5), resolution = 20,
                      n_layers = 36, n_vars = 4)

The three modes are:

Mode	What it does	When auto picks it
memory	Whole area and every dekad at once	Working set below 25% of the budget
stream	Whole area, dekads read in batches	Below the budget when streamed
tiled	Square tiles, assembled with a VRT	Otherwise, or when one layer is over 10% of the budget

The budget is half the free RAM divided by the number of future workers. Set it explicitly with options(Rwapor.memory_budget_mb = 4096).

Run a seasonal analysis

wapor_run_seasonal_analysis() plans and runs the job in one call. Leave processing at "auto", or force a mode.

crop_mask <- wapor_load_crop_mask("scheme_crop_mask.tif")

config <- list(
  period = c("2023-01-01", "2023-12-31"),
  data_source = "api",
  aeti_var = "L3-AETI-D", ret_var = "L1-RET-D", precip_var = "L1-PCP-D",
  indicators = c("agg_aeti", "etc", "adequacy_etc", "agg_peff"),
  use_crop_mask = TRUE, use_season_rasters = FALSE,
  processing = "auto",          # or "memory", "stream", "tiled"
  output_dir = "scheme_run"     # stream and tiled outputs are written here
)
params <- wapor_create_crop_params(class_value = 1, crop_name = "Maize")

result <- wapor_run_seasonal_analysis(config, params, rasters = list(crop_mask = crop_mask))
result$processing          # the plan that ran
result$coverage$aeti       # share of season days with data, per pixel

When aoi_region is not given, the analysis area is the crop mask extent.

Accuracy

Seasonal and monthly totals are summed at each source’s own resolution and resampled once onto the analysis grid. With the default nearest-neighbour resampling, every value is a raw WaPOR value or a sum of raw values. For example, AgERA5-based reference ET at about 9 km is summed on its own grid and then assigned to the 20 m pixels inside each cell. To interpolate instead:

config$resampling_method <- list(ret = "bilinear", pcp = "bilinear")

A pixel whose season is missing any dekad is NA by default rather than treated as 0 mm. Allow partial seasons with, for example, config$min_coverage <- 0.9.

Parallel tiles

Tiles run in parallel under the active future plan. terra and GDAL caches are divided between the workers automatically.

future::plan(future::multisession, workers = 4)
result <- wapor_run_seasonal_analysis(config, params, rasters = list(crop_mask = crop_mask))

Resumable tiled runs

wapor_run_seasonal_analysis_tiled() forces tiled mode and keeps a run manifest with a checksum for every tile. If a run is interrupted, rerun it with resume = TRUE: finished tiles are reused and only the missing ones are computed.

tiled <- wapor_run_seasonal_analysis_tiled(
  config, params, rasters = list(crop_mask = crop_mask),
  output_dir = "scheme_tiles", tile_size = 1024L, cog = TRUE, resume = TRUE
)
tiled$saved_files

Check your setup

inst/bench/remote_smoke_test.R runs a short live check against the WaPOR API, and inst/bench/large_raster_benchmark.R measures memory and time on synthetic data:

system.file("bench", "remote_smoke_test.R", package = "Rwapor")