Update user_pseudo_id to client_key (#181)

* Replace user_pseudo_id with client_key

* Remove stream_id from fct_ga4__user_ids

* Add descriptions for client_key throughout package

* restore stream_id to several models

* Move client_key creation to stag_ga4__events

---------

Co-authored-by: Adam Ribaudo <[email protected]>

Author: TannerHopkins

README.md

@@ -20,11 +20,11 @@ Features include:
 | stg_ga4__event_items | Contains item data associated with e-commerce events (Purchase, add to cart, etc) |
 | stg_ga4__event_to_query_string_params | Mapping between each event and any query parameters & values that were contained in the event's `page_location` field |
 | stg_ga4__user_properties | Finds the most recent occurance of specified user_properties for each user |
-| stg_ga4__derived_user_properties | Finds the most recent occurance of specific event_params value and assigns them to a user_pseudo_id. Derived user properties are specified as variables (see documentation below) |
+| stg_ga4__derived_user_properties | Finds the most recent occurance of specific event_params value and assigns them to a client_key. Derived user properties are specified as variables (see documentation below) |
 | stg_ga4__derived_session_properties | Finds the most recent occurance of specific event_params or user_properties value and assigns them to a session's session_key. Derived session properties are specified as variables (see documentation below) |
 | stg_ga4__session_conversions_daily | Produces daily counts of conversions per session. The list of conversion events to include is configurable (see documentation below) |
 | stg_ga4__sessions_traffic_sources | Finds the first source, medium, campaign, content, paid search term (from UTM tracking), and default channel grouping for each session. |
-| dim_ga4__user_pseudo_ids | Dimension table for user devices as indicated by user_pseudo_ids. Contains attributes such as first and last page viewed.| 
+| dim_ga4__client_keys | Dimension table for user devices as indicated by client_keys. Contains attributes such as first and last page viewed.| 
 | dim_ga4__sessions | Dimension table for sessions which contains useful attributes such as geography, device information, and acquisition data. Can be expensive to run on large installs (see `dim_ga4__sessions_daily`) |
 | dim_ga4__sessions_daily | Query-optimized session dimension table that is incremental and partitioned on date. Assumes that each partition is contained within a single day |
 | fct_ga4__pages | Fact table for pages which aggregates common page metrics by page_location, date, and hour. |

models/marts/core/core.yml

@@ -7,10 +7,11 @@ models:
       - name: session_key
         tests:
           - unique
-  - name: dim_ga4__user_pseudo_ids
-    description: Dimension table for user devices (user_pseudo_id) which includes data from the first and last event produced. Unique on user_pseudo_id
+  - name: dim_ga4__client_keys
+    description: Dimension table for user devices (client_key) which includes data from the first and last event produced. Unique on client_key
     columns:
-      - name: user_pseudo_id
+      - name: client_key
+        description: Hashed combination of user_pseudo_id and stream_id
         tests:
           - unique
   - name: fct_ga4__sessions
@@ -26,15 +27,16 @@ models:
         description: The total engagement time for that page_location.
       - name: avg_engagement_time_denominator
         description: Use avg_engagement_time_denominator to calculate the average engagement time, which is derived by dividing the sum of total engagement time by the product of the sum of the denominator and 1000 to get the average engagement time in seconds (average_engagement_time = sum(total_engagement_time_msec)/(sum(avg_engagement_time_denominator) *1000 )). The denominator excludes page_view events where no engagement time is recorded for the page_location within a session. However, it includes subsequent page_view events to a page_location that has previously recorded a page_view event in the same session, even if the subsequent event has no recorded engagement time.
-  - name: fct_ga4__user_pseudo_ids
-    description: Fact table with aggregate metrics at the level of the user's device (as indicated by the user_pseudo_id). Metrics are aggregated from fct_ga4__sessions.
+  - name: fct_ga4__client_keys
+    description: Fact table with aggregate metrics at the level of the user's device (as indicated by the client_key). Metrics are aggregated from fct_ga4__sessions.
     columns:
-      - name: user_pseudo_id
+      - name: client_key
+        description: Hashed combination of user_pseudo_id and stream_id
         tests: 
           - unique
   - name: fct_ga4__user_ids
-    description: Fact table with aggregate metrics at the level of the user_id when one is present, otherwise at the device level (as indicated by the user_pseudo_id). Metrics are aggregated from fct_ga4__user_pseudo_ids.
+    description: Fact table with aggregate metrics at the level of the user_id when one is present, otherwise at the device level (as indicated by the client_key). Metrics are aggregated from fct_ga4__client_keys.
     columns:
-      - name: user_id_or_user_pseudo_id
+      - name: user_id_or_client_key
         tests: 
           - unique

models/marts/core/dim_ga4__user_pseudo_ids.sql models/marts/core/dim_ga4__client_keys.sql

@@ -1,9 +1,9 @@
--- Mart for dimensions related to user devices (based on user_pseudo_id)
+-- Mart for dimensions related to user devices (based on client_key)
 
 with include_first_last_events as (
     select 
         *
-    from {{ref('stg_ga4__user_pseudo_id_first_last_events')}}
+    from {{ref('stg_ga4__client_key_first_last_events')}}
 ),
 include_first_last_page_views as (
     select 
@@ -15,19 +15,19 @@ include_first_last_page_views as (
         first_last_page_views.last_page_hostname,
         first_last_page_views.last_page_referrer
     from include_first_last_events 
-    left join {{ref('stg_ga4__user_pseudo_id_first_last_pageviews')}} as first_last_page_views using (user_pseudo_id)
+    left join {{ref('stg_ga4__client_key_first_last_pageviews')}} as first_last_page_views using (client_key)
 ),
 include_user_properties as (
 
 
 select * from include_first_last_page_views
 {% if var('derived_user_properties', false) %}
--- If derived user properties have been assigned as variables, join them on the user_pseudo_id
-left join {{ref('stg_ga4__derived_user_properties')}} using (user_pseudo_id)
+-- If derived user properties have been assigned as variables, join them on the client_key
+left join {{ref('stg_ga4__derived_user_properties')}} using (client_key)
 {% endif %}
 {% if var('user_properties', false) %}
--- If user properties have been assigned as variables, join them on the user_pseudo_id
-left join {{ref('stg_ga4__user_properties')}} using (user_pseudo_id)
+-- If user properties have been assigned as variables, join them on the client_key
+left join {{ref('stg_ga4__user_properties')}} using (client_key)
 {% endif %}
 
 )

models/marts/core/dim_ga4__sessions_daily.sql

@@ -34,7 +34,7 @@
 with event_dimensions as 
 (
     select 
-        user_pseudo_id,
+        client_key,
         session_key,
         session_partition_key,
         event_date_dt as session_partition_date,

models/marts/core/fct_ga4__user_pseudo_ids.sql models/marts/core/fct_ga4__client_keys.sql

@@ -1,5 +1,5 @@
 select
-    user_pseudo_id,
+    client_key,
     stream_id,
     min(session_start_timestamp) as first_seen_timestamp,
     min(session_start_date) as first_seen_start_date,
@@ -15,5 +15,5 @@ select
         {% endfor %}
     {% endif %}
 from {{ref('fct_ga4__sessions')}}
-group by 1,2
+group by 1, 2
 

models/marts/core/fct_ga4__pages.sql

@@ -21,8 +21,8 @@ with page_view as (
         page_title,  -- would like to move this to dim_ga4__pages but need to think how to handle page_title changing over time
         page_engagement_key,
         count(event_name) as page_views,
-        count(distinct user_pseudo_id ) as distinct_user_pseudo_ids,
-        sum( if(session_number = 1,1,0)) as new_user_pseudo_ids,
+        count(distinct client_key ) as distinct_client_keys,
+        sum( if(session_number = 1,1,0)) as new_client_keys,
         sum(entrances) as entrances,
 from {{ref('stg_ga4__event_page_view')}}
     group by 1,2,3,4,5,6,7

models/marts/core/fct_ga4__sessions.sql

@@ -1,7 +1,7 @@
 -- Stay mindful of performance/cost when leavin this model enabled. Making this model incremental on date is not possible because there's no way to create a single record per session AND partition on date. 
 
 select
-    user_pseudo_id,
+    client_key,
     session_key,
     stream_id,
     min(session_partition_min_timestamp) as session_start_timestamp,

models/marts/core/fct_ga4__sessions_daily.sql

@@ -35,7 +35,7 @@ with session_metrics as (
     select 
         session_key,
         session_partition_key,
-        user_pseudo_id,
+        client_key,
         stream_id,
         min(event_date_dt) as session_partition_date, -- Date of the session partition, does not represent the true session start date which, in GA4, can span multiple days
         min(event_timestamp) as session_partition_min_timestamp,
@@ -72,7 +72,7 @@ with session_metrics as (
     ),
     join_metrics_and_conversions as (
         select 
-            session_metrics.user_pseudo_id,
+            session_metrics.client_key,
             session_metrics.stream_id,
             session_metrics.session_partition_min_timestamp,
             session_metrics.session_partition_count_page_views,

models/marts/core/fct_ga4__user_ids.sql

@@ -1,18 +1,18 @@
 with user_id_mapped as (
     select 
-        user_pseudo_ids.*,
-        -- Use a user_id if it exists, otherwise fall back to the user_pseudo_id
-        coalesce(user_id_mapping.last_seen_user_id, user_pseudo_ids.user_pseudo_id) as user_id_or_user_pseudo_id,
-        -- Indicate whether the user_id_or_user_pseudo_id value is a user_id
+        client_keys.*,
+        -- Use a user_id if it exists, otherwise fall back to the client_key
+        coalesce(user_id_mapping.last_seen_user_id, client_keys.client_key) as user_id_or_client_key,
+        -- Indicate whether the user_id_or_client_key value is a user_id
         CASE 
             WHEN user_id_mapping.last_seen_user_id is null THEN 0 ELSE 1
         END as is_user_id
-    from {{ref('fct_ga4__user_pseudo_ids')}} user_pseudo_ids
-    left join {{ref('stg_ga4__user_id_mapping')}} user_id_mapping using (user_pseudo_id)
+    from {{ref('fct_ga4__client_keys')}} client_keys
+    left join {{ref('stg_ga4__user_id_mapping')}} user_id_mapping using (client_key)
 )
 
 select
-    user_id_or_user_pseudo_id,
+    user_id_or_client_key,
     stream_id,
     max(is_user_id) as is_user_id,
     min(first_seen_timestamp) as first_seen_timestamp,
@@ -29,5 +29,5 @@ select
         {% endfor %}
     {% endif %}
 from user_id_mapped
-group by 1,2
+group by 1, 2
 

models/staging/stg_ga4__user_pseudo_id_first_last_events.sql models/staging/stg_ga4__client_key_first_last_events.sql

@@ -4,24 +4,24 @@
 
 with first_last_event as (
     select
-        user_pseudo_id,
-        FIRST_VALUE(event_key) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS first_event,
-        LAST_VALUE(event_key) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_event,
+        client_key,
+        FIRST_VALUE(event_key) OVER (PARTITION BY client_key ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS first_event,
+        LAST_VALUE(event_key) OVER (PARTITION BY client_key ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_event,
         stream_id
     from {{ref('stg_ga4__events')}}
-    where user_pseudo_id is not null --remove users with privacy settings enabled
+    where client_key is not null --remove users with privacy settings enabled
 ),
-events_by_user_pseudo_id as (
+events_by_client_key as (
     select distinct
-        user_pseudo_id,
+        client_key,
         first_event,
         last_event,
         stream_id
     from first_last_event
 ),
 events_joined as (
     select
-        events_by_user_pseudo_id.*,
+        events_by_client_key.*,
         events_first.geo_continent as first_geo_continent,
         events_first.geo_country as first_geo_country,
         events_first.geo_region as first_geo_region,
@@ -74,11 +74,11 @@ events_joined as (
         events_last.user_campaign as last_user_campaign,
         events_last.user_medium as last_user_medium,
         events_last.user_source as last_user_source,
-    from events_by_user_pseudo_id
+    from events_by_client_key
     left join {{ref('stg_ga4__events')}} events_first
-        on events_by_user_pseudo_id.first_event = events_first.event_key
+        on events_by_client_key.first_event = events_first.event_key
     left join {{ref('stg_ga4__events')}} events_last
-        on events_by_user_pseudo_id.last_event = events_last.event_key
+        on events_by_client_key.last_event = events_last.event_key
 )
 
 select * from events_joined

models/staging/stg_ga4__user_pseudo_id_first_last_events.yml models/staging/stg_ga4__client_key_first_last_events.yml

@@ -1,9 +1,10 @@
 version: 2
 
 models:  
-  - name: stg_ga4__user_pseudo_id_first_last_events
+  - name: stg_ga4__client_key_first_last_events
     description: Captures the first and last event completed by the user's device in order to pull in the first and last geo, device, and traffic source seen from the user
     columns:
-      - name: user_pseudo_id
+      - name: client_key
+        description: Hashed combination of user_pseudo_id and stream_id
         tests: 
           - unique

models/staging/stg_ga4__user_pseudo_id_first_last_pageviews.sql models/staging/stg_ga4__client_key_first_last_pageviews.sql

@@ -4,34 +4,34 @@
 
 with page_views_first_last as (
     select
-        user_pseudo_id,
-        FIRST_VALUE(event_key) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS first_page_view_event_key,
-        LAST_VALUE(event_key) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_page_view_event_key
+        client_key,
+        FIRST_VALUE(event_key) OVER (PARTITION BY client_key ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS first_page_view_event_key,
+        LAST_VALUE(event_key) OVER (PARTITION BY client_key ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_page_view_event_key
     from {{ref('stg_ga4__event_page_view')}}
-    where user_pseudo_id is not null -- Remove users with privacy settings enabled
+    where client_key is not null -- Remove users with privacy settings enabled
 ),
-page_views_by_user_pseudo_id as (
+page_views_by_client_key as (
     select distinct
-        user_pseudo_id,
+        client_key,
         first_page_view_event_key,
         last_page_view_event_key
     from page_views_first_last
 ),
 
 page_views_joined as (
     select
-        page_views_by_user_pseudo_id.*,
+        page_views_by_client_key.*,
         first_page_view.page_location as first_page_location,
         first_page_view.page_hostname as first_page_hostname,
         first_page_view.page_referrer as first_page_referrer, 
         last_page_view.page_location as last_page_location,
         last_page_view.page_hostname as last_page_hostname,
         last_page_view.page_referrer as last_page_referrer
-    from page_views_by_user_pseudo_id
+    from page_views_by_client_key
     left join {{ref('stg_ga4__event_page_view')}} first_page_view
-        on page_views_by_user_pseudo_id.first_page_view_event_key = first_page_view.event_key
+        on page_views_by_client_key.first_page_view_event_key = first_page_view.event_key
     left join {{ref('stg_ga4__event_page_view')}} last_page_view
-        on page_views_by_user_pseudo_id.last_page_view_event_key = last_page_view.event_key
+        on page_views_by_client_key.last_page_view_event_key = last_page_view.event_key
 )
 
 select * from page_views_joined

models/staging/stg_ga4__client_key_first_last_pageviews.yml

@@ -0,0 +1,10 @@
+version: 2
+
+models:  
+  - name: stg_ga4__client_key_first_last_pageviews
+    description: Captures data related to the first and last page view that each user device has completed (by client_key).
+    columns:
+      - name: client_key
+        description: Hashed combination of user_pseudo_id and stream_id
+        tests:
+          - unique

models/staging/stg_ga4__derived_user_properties.sql

@@ -3,15 +3,15 @@
   materialized = "table"
 ) }}
 
--- Remove null user_pseudo_id (users with privacy enabled)
+-- Remove null client_key (users with privacy enabled)
 with events_from_valid_users as (
     select * from {{ref('stg_ga4__events')}}
-    where user_pseudo_id is not null
+    where client_key is not null
 ),
 unnest_user_properties as
 (
     select 
-        user_pseudo_id,
+        client_key,
         event_timestamp
         {% for up in var('derived_user_properties', []) %}
             ,{{ ga4.unnest_key('event_params',  up.event_parameter ,  up.value_type ) }}
@@ -20,9 +20,9 @@ unnest_user_properties as
 )
 
 SELECT DISTINCT
-    user_pseudo_id
+    client_key
     {% for up in var('derived_user_properties', []) %}
         , LAST_VALUE({{ up.event_parameter }} IGNORE NULLS) OVER (user_window) AS {{ up.user_property_name }}
     {% endfor %}
 FROM unnest_user_properties
-WINDOW user_window AS (PARTITION BY user_pseudo_id ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
+WINDOW user_window AS (PARTITION BY client_key ORDER BY event_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)

models/staging/stg_ga4__derived_user_properties.yml

@@ -2,8 +2,9 @@ version: 2
 
 models:  
   - name: stg_ga4__derived_user_properties
-    description: Optional model that will pull out the most recent instance of a particular event parameter for each device (user_pseudo_id). Later used in the dim_ga4__user_pseudo_id dimension table.
+    description: Optional model that will pull out the most recent instance of a particular event parameter for each device (client_key). Later used in the dim_ga4__client_key dimension table.
     columns:
-      - name: user_pseudo_id
+      - name: client_key
+        description: Hashed combination of user_pseudo_id and stream_id
         tests:
           - unique  

models/staging/stg_ga4__events.sql

@@ -7,12 +7,18 @@ with base_events as (
     select * from {{ref('base_ga4__events_intraday')}}
     {% endif %}
 ),
--- Add unique key for sessions. session_key will be null if user_pseudo_id is null due to consent being denied. ga_session_id may be null during audience trigger events. 
+-- Add key that captures a combination of stream_id and user_pseudo_id to uniquely identify a 'client' (aka. a device) within a single stream
+include_client_key as (
+    select *
+    , to_base64(md5(concat(user_pseudo_id, stream_id))) as client_key
+    from base_events
+),
+-- Add key for sessions. session_key will be null if client_key is null due to consent being denied. ga_session_id may be null during audience trigger events. 
 include_session_key as (
     select 
         *,
-        to_base64(md5(CONCAT(stream_id, user_pseudo_id, CAST(session_id as STRING)))) as session_key -- Surrogate key to determine unique session across streams and users. Sessions do NOT reset after midnight in GA4
-    from base_events
+        to_base64(md5(CONCAT(client_key, CAST(session_id as STRING)))) as session_key
+    from include_client_key
 ),
 -- Add a key that combines session key and date. Useful when working with session table within date-partitioned tables
 include_session_partition_key as (

models/staging/stg_ga4__events.yml

@@ -4,6 +4,8 @@ models:
   - name: stg_ga4__events
     description: Staging model that generates keys for users, sessions, and events. Also parses URLs to remove query string params as defined in project config. 
     columns:
+      - name: client_key
+        description: Surrogate key created from stream_id and user_pseudo_id. Provides a way to uniquely identify a user's device within a stream. Important when using the package to combine data across properties and streams.
       - name: event_key
         tests:
           - unique