(Demo) Fix: customer lifetime value calculation in customers

[even-wei]

👋 This is a demo pull request of dbt model changes, powered by Recce to provide proof of correctness during development and change review.
This is still early and we'd love your feedback!

Description & motivation

The customer lifetime value (CLV) in customers model incorrectly included orders that were not yet completed, leading to potentially inaccurate business insights.
I modified the CLV calculation within customers to consider only "completed" orders.

To-do before merge

Notify Stakeholders

• Ask Sarah to review the customer segment impacts on marketing (see impact section below)

Lineage:

Lineage DAG Diff

Modified customers model to adjust customer_lifetime_value calculation.

 

---

Validation of models:

As expected, the customer lifetime value has changed. The average value of CLV has been reduced and this affects 98% of customers.

Query Diff: customers

The customer_lifetime_value has reduced for certain customers. This is expected given the change to the CLV calculation.

 

---

Profile Diff: customers

Min/max remains unchanged, average value has dropped from 2758.6 to 1871.76 as expected.

 

---

Value Diff: customers

Only 1.19% match on customer_lifetime_value.

 

---

Magic metric of customers

Our magic metric, average lifetime value, goes down as well.

SQL

SELECT
    DATE_TRUNC('week', first_order) AS first_order_week,
    AVG(customer_lifetime_value) AS avg_lifetime_value
FROM
    {{ ref("customers") }}
WHERE first_order is not NULL    
GROUP BY
    first_order_week
ORDER BY
    first_order_week;

Impact considerations:

The adjustment to CLV values has resulted in 26.45% change of customer ‘value segment’. This impact is also expected.

Value Diff: customer_segments

customer_lifetime_value as expected matches parent table at 1.19%.
value_segment is changed by 26.45%.

 

---

Query Diff: customer_segments

value_segment has changed for some of those customers with a decreased CLV. This is expected and I'll notify the business team.

---

Top-k diff: value_segment

We've seen a 17% drop in high-value customers. We need to discuss with stakeholders whether an adjustment to the threshold is necessary.

There is no change in customer_order_pattern.

Value diff: customer_order_pattern

There is no change.

Changes to existing models:

N/A

Checklist:

This checklist is mostly useful as a reminder of small things that can easily be
forgotten – it is meant as a helpful tool rather than hoops to jump through.
Put an x in all the items that apply, make notes next to any that haven't been
addressed, and remove any items that are not relevant to this PR.

• My pull request represents one logical piece of work.
• My commits are related to the pull request and look clean.
• My SQL follows the style guide.
• I have materialized my models appropriately.
• I have added appropriate tests and documentation to any new models.
• I have updated the README file.

[clkao]

Should we also check customer_order_pattern?

[even-wei]

There is no change in customer_order_pattern.
I will add it and update the checklist.

[even-wei]

Since the sources have been updated in the production environment, I've rebased the PR, rerun the checks, and updated the body. Most of the checks yield the same results, but I noticed that 99% of the value segments are classified as 'high value' (refer to the Top-k diff above). I plan to review the threshold next week.

[github-actions]

Recce Summary

Manifest Information

	Manifest	Catalog
Base	2024-09-20 09:04:40	2024-09-20 09:04:40
Current	2024-09-20 09:05:31	2024-09-20 09:05:32

Lineage Graph

graph LR
model.jaffle_shop.customers["customers

[What's Changed]
Code, Value Diff"]
style model.jaffle_shop.customers stroke:#ffa502
model.jaffle_shop.customers---->model.jaffle_shop.customer_segments
model.jaffle_shop.customers---->model.jaffle_shop.customer_order_pattern
model.jaffle_shop.customer_segments["customer_segments"]
model.jaffle_shop.customer_order_pattern["customer_order_pattern"]

Loading

Checks Summary

Checks Run	Data Mismatch Detected
4	2

Checks of Data Mismatch Detected

Name	Type	Mismatched Nodes
Value diff of customers	Value Diff	customers
Query diff of customers avg lifetime value	Query Diff	N/A

See PR page: https://cloud.datarecce.io/DataRecce/jaffle_shop_duckdb/pulls/1

How to check the recce result

# Launch the recce server based on the state file
recce server --review --cloud --password yummy_jaffle

# Open the recce server http://localhost:8000 by your browser