What are the Benefits of having separate layer/a dedicated schema for each layer in ETL (extract , transform and load) ?

Hello Experts,

what are possible benefits of having separate layer/a dedicated schema for each layer in ETL (extract , transform and load), I mean one dedicated layer for staging , one for type 1 persistent tables and a dedicated schema/layer for tables at dimensional model ? why it is not recommended to have all tables in one schema?

Thanks,

Rajneesh

rajneesh4u

posted on 27 Apr 20

Enjoy great content like this and a lot more !

Signup for a free account to write a post / comment / upvote posts. Its simple and takes less than 5 seconds




nVector27-Apr-20

In a typical cloud data warehouse, For batch inserts, There are three minimum layers:

1. Cloud Storage Buckets - Where the files land

2. Staging Layer - Where the files are inserted into tables. (1:1 mapping)

3. Target Layer - Where the Dimensions and Facts are stored

The CDC (Change data capture) happens between Staging and Target Layer.

Why do we need separate layers ? For audit and control

1. We can easily restrict the access with multiple layers (Your business users may get confused if you expose them to staging layers)

2. Its easier for backups (we can only take backups of Target tables and ignore the Stage layer)

3. Its easier to purge (Stage tables are kill and fill, so everyone knows its safe to delete the data in the stage tables)

4. Its much simpler to have a staging database, instead of having two similarly named tables in the target layer 

rajneesh4u28-Apr-20

Thanks, it helps !!!