Functions Staging (data)

1 functions

1.1 consolidation
1.2 alignment
1.3 minimizing contention
1.4 independent scheduling/multiple targets
1.5 change detection
1.6 cleansing data
1.7 aggregate precalculation
1.8 data archiving , troubleshooting

functions

staging areas can designed provide many benefits, primary motivations use increase efficiency of etl processes, ensure data integrity , support data quality operations. functions of staging area include following:

consolidation

one of primary functions performed staging area consolidation of data multiple source systems. in performing function staging area acts large bucket in data multiple source systems can temporarily placed further processing. common tag data in staging area additional metadata indicating source of origin , timestamps indicating when data placed in staging area.

alignment

aligning data includes standardization of reference data across multiple source systems , validation of relationships between records , data elements different sources. data alignment in staging area function closely related to, , acting in support of, master data management capabilities.

minimizing contention

the staging area , etl processes supports designed goal of minimizing contention within source systems. copying required data source systems staging area in 1 shot more efficient retrieving individual records (or small sets of records) on one-off basis. former method takes advantage of technical efficiencies, such data streaming technologies, reduced overhead through minimizing need break , re-establish connections source systems , optimization of concurrency lock management on multi-user source systems. copying source data source systems , waiting perform intensive processing , transformation in staging area, etl process exercises great degree of control on concurrency issues during processing.

independent scheduling/multiple targets

the staging area can support hosting of data processed on independent schedules, , data meant directed multiple targets. in instances data might pulled staging area @ different times held , processed @ once. situation might occur when enterprise processing done across multiple time zones each night, instance. in other cases data might brought staging area processed @ drent times; or staging area may used push data multiple target systems. example, daily operational data might pushed operational data store (ods) while same data may sent in monthly aggregated form data warehouse.

change detection

the staging area supports efficient change detection operations against target systems. functionality particularly useful when source systems not support reliable forms of change detection, such system-enforced timestamping, change tracking or change data capture (cdc).

cleansing data

data cleansing includes identification , removal (or update) of invalid data source systems. etl process utilizing staging area can used implement business logic identify , handle invalid data. invalid data defined through combination of business rules , technical limitations. technical constraints may additionally placed on staging area structures (such table constraints in relational database) enforce data validity rules.

aggregate precalculation

precalculation of aggregates, complex calculations , application of complex business logic may done in staging area support highly responsive service level agreements (slas) summary reporting in target systems.

data archiving , troubleshooting

data archiving can performed in, or supported by, staging area. in scenario staging area can used maintain historical records during load process, or can used push data target archive structure. additionally data may maintained within staging area extended periods of time support technical troubleshooting of etl process....