Release 1.0 FOCUS update (#352)

* Added initial documentation to the README Signed-off-by: Varun Mittal <[email protected]> * Added more details in README about initial service design. Added contributing rules Signed-off-by: Mike Fuller ([email protected]) Signed-off-by: Varun Mittal <[email protected]> * Initial code import. Signed-off-by: Varun Mittal <[email protected]> * Added data generators. Signed-off-by: Varun Mittal <[email protected]> * Imported code into focus_converter_base. Signed-off-by: Varun Mittal <[email protected]> * Updated git repository address. Signed-off-by: Varun Mittal <[email protected]> * Lint: Fixed step 4 command in README.md. Signed-off-by: Varun Mittal <[email protected]> * Updated README.md to include windows powershell instructions. Signed-off-by: Varun Mittal <[email protected]> * Added file_names to base_config and added file_names to visualization layer. Signed-off-by: Varun Mittal <[email protected]> * Added jupyter notebook to show plan graphically. Signed-off-by: Varun Mittal <[email protected]> * Added sql validation while building plan. (#49) Signed-off-by: Varun Mittal <[email protected]> * Fixed test data generators to use if/else instead of match statement. (#43) Signed-off-by: Varun Mittal <[email protected]> * Used if-else for backward compatibility and added unittests for all python versions. (#36) Signed-off-by: Varun Mittal <[email protected]> * Added exception handler in convert function to allow cleanup of exporter processes. (#46) Signed-off-by: Varun Mittal <[email protected]> * Updated gcp ChargePeriodStart and ChargePeriodEnd to use date types as UTC datetime format. (#37) Signed-off-by: Varun Mittal <[email protected]> * Added config for chargetype for AWS. (#42) Signed-off-by: Varun Mittal <[email protected]> * Added gcp cost type config. (#48) Signed-off-by: Varun Mittal <[email protected]> * Fixed credits.amount by adding field schema field type in case it is missing. (#104) Signed-off-by: Varun Mittal <[email protected]> * Add datetime function to convert to ms for datetime conversion functions. (#107) Signed-off-by: Varun Mittal <[email protected]> * Added capability to specify column prefix in the source dataset. (#105) Signed-off-by: Varun Mittal <[email protected]> * Implemented lookup operation for converting column values using a reference dataset. (#117) Signed-off-by: Varun Mittal <[email protected]> * Fixed sql execution by creating unique sql context for each query. (#115) Signed-off-by: Varun Mittal <[email protected]> * Added value map function to allow values maps with defaults. (#112) Signed-off-by: Varun Mittal <[email protected]> * Added base function to create static value assignment. (#109) Signed-off-by: Varun Mittal <[email protected]> * Added aws provider conversion config. (#110) Signed-off-by: Varun Mittal <[email protected]> * Lookup fixes (#122) * Added test notebook glob pattern to .gitignore. * Added reference path check to differentiate between package file vs test file. Signed-off-by: Varun Mittal <[email protected]> * Implemented new unnest logic using pl.col.struct expression implementing aggregation logic for nested list values. (#121) Signed-off-by: Varun Mittal <[email protected]> * Added service category mappings for AWS. (#114) Signed-off-by: Varun Mittal <[email protected]> * Added gcp service category mapping using lookup operation. (#123) Signed-off-by: Varun Mittal <[email protected]> * Added documentation for different conversion operations supported with examples. (#124) Signed-off-by: Varun Mittal <[email protected]> * Updated main to allow running validator and added sample notebook. (#119) Signed-off-by: Varun Mittal <[email protected]> * Added focus column names to config validator. (#128) Signed-off-by: Varun Mittal <[email protected]> * Added focus column enum for SubAccountId. (#129) Signed-off-by: Varun Mittal <[email protected]> * Lint: Fixed black and isort errors. Also python 3.8 import errors. (#127) Signed-off-by: Varun Mittal <[email protected]> * Fixed Billing Currency focus enum. (#130) Signed-off-by: Varun Mittal <[email protected]> * Temporarily disable python 3.12 checks as pyarrow not compatible until 14.0 release. (#133) Signed-off-by: Varun Mittal <[email protected]> * Added azure dimension configs. (#132) Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Cory Syvenky <[email protected]> Add Azure provider field (#135) Signed-off-by: Cory Syvenky <[email protected]> Fix dimension order for provider (#136) Signed-off-by: Cory Syvenky <[email protected]> Add Azure ServiceCategory (#137) * Add Azure ServiceCategory Signed-off-by: Cory Syvenky <[email protected]> Added notebook cell for Azure plan visualization. (#138) Signed-off-by: Cory Syvenky <[email protected]> Updated README to include instructions for direct pip install. (#131) Signed-off-by: Varun Mittal <[email protected]> Added OCI Provider dimension (#85) (#139) Signed-off-by: Irena Jurica <[email protected]> Simple Typo fix in readme Signed-off-by: Mike Fuller <[email protected]> * Added OCI Region dimension (#86) (#148) Signed-off-by: ijurica <[email protected]> * Added OCI Billing Account Id dimension (#29) (#147) Signed-off-by: ijurica <[email protected]> * add resourcename field from azure config (#145) Signed-off-by: Cory Syvenky <[email protected]> * add billingcurrency field from azure config (#144) Signed-off-by: Cory Syvenky <[email protected]> * converted all date fields to datetime (#143) Signed-off-by: Cory Syvenky <[email protected]> * Added AvailibityZone, BillingCurrency, InvoiceIssuer, ResourceId, ServiceName for AWS. (#140) Signed-off-by: Varun Mittal <[email protected]> * Added image path parameter for explain command to save graph. (#149) --------- Signed-off-by: Varun Mittal <[email protected]> * Fixed datetime columns imported as str instead of datetime[ns]. (#150) Signed-off-by: Varun Mittal <[email protected]> * Added base functions for validating column names before starting conversion. (#151) Signed-off-by: Varun Mittal <[email protected]> * Added new conversion plan to add default null values if a column is missing in source dataset. (#158) Signed-off-by: Varun Mittal <[email protected]> * Added ignore_errors while reading cvs to allow reading even if the csv parsing fails. (#157) Signed-off-by: Varun Mittal <[email protected]> * Shuffle docs around (#156) Signed-off-by: Joshua Kwan <[email protected]> Co-authored-by: Varun Mittal <[email protected]> Co-authored-by: Mike Fuller <[email protected]> * Aws 1.0 transform plans along with more sql validations. (#154) Signed-off-by: Varun Mittal <[email protected]> * addition of an anonymized AWS CUR dataset (#155) Signed-off-by: Cory Syvenky <[email protected]> * Added multiple OCI focus v0.5 dimensions: AvailabilityZone (issue #82), BillingCurrency (issue #83), ResourceId (issue #56), ServiceName (issue #88), SubAccountId (issue #89) (#163) Signed-off-by: ijurica <[email protected]> * Added OCI BilledCost (issue #35) and EffectiveCost (issue #90) (#162) Signed-off-by: ijurica <[email protected]> * chore: fix a typo on aws category mapping filename (#160) Signed-off-by: Ivan Stoiev <[email protected]> * Added GCP configs for Provider,AvailabilityZone,BillingCurrency,ResourceId (#153) Signed-off-by: Varun Mittal <[email protected]> * Dependency fixes + readme edit (#165) Signed-off-by: Barak Freiman <[email protected]> * Create ChargePeriodStart FOCUS from Billing Start (#239) Signed-off-by: Cory Syvenky <[email protected]> * addition of Publisher field for Azure provider. (#237) Signed-off-by: Cory Syvenky <[email protected]> * addition of BilledCost field (#236) Signed-off-by: Cory Syvenky <[email protected]> * addition of Billing Account Name and Billing Account ID fields for Azure. (#235) Signed-off-by: Cory Syvenky <[email protected]> * aws:cur as a new provider for processing AWS CUR CSV (#240) Signed-off-by: spriharani <[email protected]> * Fixed python dependencies. (#164) Added jupyterlab as optional dependency and removed redundant dependencies. Signed-off-by: Varun Mittal <[email protected]> * Renamed aws:cur to aws-cur to avoid path issues on windows. (#241) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.0 → 0.7.1-dev1, Init pypi build (#242) Signed-off-by: Varun Mittal <[email protected]> * Hotfix: Temporarily change focus-validator version to support pip. (#243) Signed-off-by: Varun Mittal <[email protected]> * Handle lineItem/ResourceId if it is missing in source data (#246) Signed-off-by: spriharani <[email protected]> * fix(aws-cur): wrong aws cur column used on ChargeSubcategory calculation (#244) Signed-off-by: Ivan Stoiev <[email protected]> * fix(aws-cur): wrong invoice issuer field calculation (#245) Signed-off-by: Ivan Stoiev <[email protected]> * Added specific config checks for aws,azure and oci. Also added dtype re-enforcements for datatime/date. (#247) Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Ivan <[email protected]> * Added cli utility to export conversion rules to a csv/markdown per provider. (#249) Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> * Bump version: 0.7.1-dev1 → 0.7.1-dev2 (#250) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev2 → 0.7.1-dev3 * Fix aws-cur - parse bill/PayerAccountId as string (#251) Signed-off-by: spriharani <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added focus_validation integration. (#254) Signed-off-by: Varun Mittal <[email protected]> * Updated conversion config to include feature request, column conversion logic and bug_report. (#255) Signed-off-by: Varun Mittal <[email protected]> * Added FOCUS columns to output types to produce empty columns if conversion plan not available yet. (#256) Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> * Standardize AWS category mapping, similar to Azure and GCP (#259) Signed-off-by: Brian Wyka <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added converter function to compute start and end of datetime column. (#248) Added configs for OCI using new start/end function to compute BillingPeriodStart, BillingPeriodEnd. Fixes #30, #31, #32, #33. Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added conversion config for GCP SkuId, resoloving #232. (#260) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Adding a performance profiling tool and applying to existing tests (#266) Signed-off-by: Justin Ohrenberger <[email protected]> * Adding flake8 config for higher quality code. (#268) Signed-off-by: Justin Ohrenberger <[email protected]> * Removing Azure D011 as it appears to be a duplicate of D007 (only with an error - appears to be trying to convert SubscriptionID to BillingCurrency) * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of availability zone field (#273) Signed-off-by: Cory Syvenky <[email protected]> * addition of EffectiveCost field. (#271) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Azure usage quantity conversion yaml file (#276) Signed-off-by: David Flood <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of focus column ChargeDescription (#290) Signed-off-by: Cory Syvenky <[email protected]> * Changed orientation of generated UML diagram to left-right from top-down (#289) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of focus column PricingCategory (#295) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * addition of focus column ChargeFrequency (#291) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of ChargeSubcategory (#296) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Bug fix for effective cost (#292) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added definition for CommitmentDiscountName. Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Azure commitment discount name conversion file (#300) Signed-off-by: David Flood <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Addition of InvoiceIssuer FOCUS field. (#308) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Addition of UsageUnit FOCUS field via Pricing Units mapping. (#309) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Addition of SkuPriceId FOCUS field, defaulting to null. (#310) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Reducing cyclomatic complexity (#306) Signed-off-by: Justin Ohrenberger <[email protected]> * Add docker image. (#311) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev3 → 0.7.1-dev4 Signed-off-by: Varun Mittal <[email protected]> * Upgraded polars(0.19.2 -> 0.20.6) to match sql aggregate functions. Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev4 → 0.7.1-dev5 Signed-off-by: Varun Mittal <[email protected]> * Add progress bar for each provider. (#314) Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> * Azure ResourceId conversion (#301) Signed-off-by: David Flood <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * remove-dupe: Remove duplicate Azure microsoft.security mapping (#315) Signed-off-by: Brian Wyka <[email protected]> * addition of CommitmentDiscountId (#298) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Gcp new column configs. (#317) * Added configs for PricingQuantity(#212), SkuPriceId(#227), UsageQuantity(#223), UsageUnit(#221) for GCP. Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of CommitmentDiscountType (#299) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of CommitmentDiscountCategory (#297) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added string conversion plans to support lower, upper, title case transforms. (#319) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Fixed docker metadata tag to include current version tags. (#320) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev5 → 0.7.1-dev6 * addition of ChargeCategory column. (#322) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added conversion configs for OCI. (#303) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added new plan to split string and get part using index. (#323) Signed-off-by: Varun Mittal <[email protected]> * Removed BilledCurrency definition. (#324) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added new AWS dimension configs for ResourceName(#51), ResourceName(#278), Publisher(#9). (#326) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Fixed pie-chart import for progress/README.md Signed-off-by: Varun Mittal <[email protected]> * Added conversion configs for aws-cur provider(AWS Cur CSV provider). (#327) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added conversion step to export FOCUS data. (#328) Signed-off-by: Varun Mittal <[email protected]> * Added step to convert azure sample data to FOCUS and add to artifact. Signed-off-by: Varun Mittal <[email protected]> * Moved jupyter dependencies to optional, allowing to install --with=jupyter. (#329) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev6 → 0.7.1-dev7 * Added PricingCategory for AWS (#284). (#330) * Added PricingCategory for AWS (#284). Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added input data auto sening module to simplify source data inputs (#332) Added input data auto sening module to simplify source data inputs Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev7 → 0.7.1-dev8 * fix(aws): Region column source (#333) Signed-off-by: Ivan Stoiev <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * aws-category: Add ComputeSavingsPlans to aws category mapping (#334) Signed-off-by: Brian Wyka <[email protected]> * Adding Amazon Bedrock entry (#335) Signed-off-by: varangian-core <[email protected]> * Output specific parquet filename (#339) Signed-off-by: David Sschneider <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Fix multiprocessing queue and downgraded polars version temporarily until pyarrow compatibility is restored. (#341) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added GCP missing dimensions for EffectiveCost(#168), ServiceName(#77), ResourceName(#52), PricingUnit(#217), ListCost(#205) (#340) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Order focus exported columns to make it easier to read and parse (#344) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev8 → 0.7.1-dev9 Signed-off-by: Varun Mittal <[email protected]> * Added support for file CSV export (#349) * Added support for file CSV export Signed-off-by: Mike Fuller <[email protected]> * Fix import sorting Signed-off-by: Mike Fuller <[email protected]> * Fix up defaults for class to pass tests Signed-off-by: Mike Fuller <[email protected]> --------- Signed-off-by: Mike Fuller <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Updated column configs to align better with 1.0 (GA) spec Fixed tests to work with updated columns Signed-off-by: Mike Fuller <[email protected]> * Bump version to assist build Signed-off-by: Mike Fuller <[email protected]> * Bump version: 0.7.2-dev1 → 1.0.0-dev1 Signed-off-by: Mike Fuller <[email protected]> * Bump version: 1.0.0-dev1 → 1.0.0 Signed-off-by: Mike Fuller <[email protected]> --------- Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: Mike Fuller ([email protected]) Signed-off-by: Mike Fuller <[email protected]> Signed-off-by: ijurica <[email protected]> Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Joshua Kwan <[email protected]> Signed-off-by: Ivan Stoiev <[email protected]> Signed-off-by: Barak Freiman <[email protected]> Signed-off-by: spriharani <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Signed-off-by: spriharani <[email protected]> Signed-off-by: Brian Wyka <[email protected]> Signed-off-by: Justin Ohrenberger <[email protected]> Signed-off-by: David Flood <[email protected]> Signed-off-by: varangian-core <[email protected]> Signed-off-by: David Sschneider <[email protected]> Co-authored-by: Varun Mittal <[email protected]> Co-authored-by: Irena Jurica <[email protected]> Co-authored-by: Cory Syvenky <[email protected]> Co-authored-by: Joshua Kwan <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: Barak <[email protected]> Co-authored-by: spriharani <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> Co-authored-by: Brian Wyka <[email protected]> Co-authored-by: johrenberger <[email protected]> Co-authored-by: David Flood <[email protected]> Co-authored-by: David Flood <[email protected]> Co-authored-by: Akari <[email protected]> Co-authored-by: David Schneider (Ollion) <[email protected]>
finopsfoundation · Jun 6, 2024 · bbfdb12 · bbfdb12
1 parent ea00d37
commit bbfdb12
Show file tree

Hide file tree

Showing 301 changed files with 14,944 additions and 46 deletions.
diff --git a/.github/ISSUE_TEMPLATE/add-modify-conversion-logic.md b/.github/ISSUE_TEMPLATE/add-modify-conversion-logic.md
@@ -0,0 +1,20 @@
+---
+name: Add/Modify conversion config
+about: Community proposal to add / enhance a column conversion config for a provider
+title: "[Proposal] Title for proposal"
+labels: proposal
+assignees: ''
+
+---
+
+### Provider
+> Provider type (e.g. AWS, GCP etc.)
+
+### Column
+> Column name
+
+### A SQL query or example steps to transform
+
+
+### Context / Supporting information
+>Description of the conversion logic
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,39 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Load a sample cost dataset as provider
+2. Run conversion
+3. If runs but produces incorrect result, run sql query on dataset using
+```
+import polars as pl
+
+lf = pl.scan_parquet(<path_to_converted_data>)
+sql_context = SQLFunctions.create_sql_context(lf=lf)
+df = sql_context.execute(sql_query, eager=False).collect()
+print(df.select(["column1", ...]).limit(10)
+```
+
+**Expected behavior**
+Value format/value type.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Converter version [e.g. 0.7]
+ - Focus Spec Version [e.g. 1.0]
+
+**Additional context**
+Add any other context about the problem here.
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
diff --git a/.github/workflows/docker_build.yaml b/.github/workflows/docker_build.yaml
@@ -0,0 +1,39 @@
+name: Docker Build
+on:
+  create:
+    tags:
+      - 'v\d\.\d\.\d'
+      - 'v\d\.\d\.\d-(dev|rc)\d'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            finopsfoundation/focus-converter
+          tags: |
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}
+      - name: Build and Push Versioned Docker Image
+        id: build-and-push
+        uses: docker/build-push-action@v5
+        with:
+          context: ./
+          file: ./Dockerfile
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
diff --git a/.github/workflows/export_conversion_rules.yaml b/.github/workflows/export_conversion_rules.yaml
@@ -0,0 +1,61 @@
+name: Export Conversion Rules
+on:
+  push:
+    branches:
+      - main
+      - dev
+      - issue/**
+
+jobs:
+  build:
+    name: Export Conversion Rules
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Copy converter_base to current directory Linux, MacOS
+        run: |
+          cp -r focus_converter_base/* ./
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+      - name: Install poetry
+        uses: abatilo/actions-poetry@v2
+      - name: Setup a local virtual environment
+        run: |
+          poetry config virtualenvs.create true --local
+          poetry config virtualenvs.in-project true --local
+          poetry lock
+      - uses: actions/cache@v3
+        name: Define a cache for the virtual environment based on the dependencies lock file
+        with:
+          path: ./.venv
+          key: venv-${{ hashFiles('poetry.lock') }}
+      - name: Install dependencies
+        run: |
+          poetry install
+      - name: Create conversion rule export directory
+        run: |
+          mkdir -p conversion_rules_export/{markdown,csv}
+          mkdir -p progress
+          mkdir exported_sample_data
+      - name: Generate rule export
+        run: |
+          poetry run focus-export-conversion-rules  --output-dir conversion_rules_export/markdown --output-format markdown
+          poetry run focus-export-conversion-rules  --output-dir conversion_rules_export/csv --output-format csv
+      - name: Generate progress pie charts
+        run: |
+          poetry run focus-generate-progress-pie-chart  --output-dir progress/ --csv-rules-export-dir conversion_rules_export/csv
+      - name: Generate converted sample data
+        run: |
+          poetry run focus-export-converted-sample-data --output-dir exported_sample_data
+      - name: Upload converted data to artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: exported-sample-data
+          path: exported_sample_data
+      - name: Add & Commit
+        uses: EndBug/[email protected]
+        with:
+          add : "['conversion_rules_export', 'progress']"
+          author_name: FinOps Converter Bot
+          author_email: [email protected]
+          commit: --signoff
diff --git a/.github/workflows/focus_validation.yaml b/.github/workflows/focus_validation.yaml
@@ -0,0 +1,75 @@
+name: Validate Focus Data
+on:
+  push:
+    branches:
+      - main
+      - dev
+      - issue/**
+permissions:
+  contents: read
+  actions: read
+  checks: write
+jobs:
+  validate_focus:
+    name: "Run Validator - ${{ matrix.providers.name }}"
+    strategy:
+      matrix:
+        providers:
+          - name: aws-cur
+            sample_dataset: tests/provider_config_tests/aws/sample-anonymous-aws-export-dataset.csv
+            data_format: csv
+          - name: azure
+            sample_dataset: tests/provider_config_tests/azure/sample-anonymous-ea-export-dataset.csv
+            data_format: csv
+          - name: oci
+            sample_dataset: tests/provider_config_tests/oci/reports_cost-csv_0000000030000269.csv
+            data_format: csv
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v3
+      - name: Copy converter_base to current directory Linux, MacOS
+        run: |
+          cp -r focus_converter_base/* ./
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.9
+      - name: Install poetry
+        uses: abatilo/actions-poetry@v2
+      - name: Setup a local virtual environment
+        run: |
+          poetry config virtualenvs.create true --local
+          poetry config virtualenvs.in-project true --local
+          poetry lock
+      - uses: actions/cache@v3
+        name: Define a cache for the virtual environment based on the dependencies lock file
+        with:
+          path: ./.venv
+          key: venv-${{ hashFiles('poetry.lock') }}
+      - name: Install dependencies
+        run: |
+          poetry install
+      - name: Convert sample dataset
+        run: |
+          poetry run focus-converter convert \
+            --data-path ${{ matrix.providers.sample_dataset }} \
+            --export-path /tmp/${{ matrix.providers.name }}/ \
+            --provider ${{ matrix.providers.name }} \
+            --data-format ${{ matrix.providers.data_format }}
+      - name: super-step
+        shell: sh
+        run: |
+          mkdir -p reports
+          poetry run focus-validator \
+            --data-file /tmp/${{ matrix.providers.name }}//*.parquet \
+            --output-type unittest \
+            --output-destination reports/focus_tests_${{ matrix.providers.name }}.xml
+      - name: FOCUS Validation Report-2 ${{ matrix.providers.name }}
+        uses: dorny/test-reporter@v1
+        if: success() || failure()    # run this step even if previous step failed
+        continue-on-error: true
+        with:
+          name: validate_focus-test-results-${{ matrix.providers.name }}
+          path: reports/focus_tests_${{ matrix.providers.name }}.xml
+          reporter: java-junit
diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml
@@ -0,0 +1,48 @@
+name: Lint
+
+on:
+  push:
+  pull_request:
+    branches:
+      - main
+      - dev
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Copy converter_base to current directory Windows
+        if: ${{ matrix.os == 'windows-latest' }}
+        run: |
+          xcopy /E /I /Y focus_converter_base\* .\
+      - name: Copy converter_base to current directory Linux, MacOS
+        if: ${{ matrix.os != 'windows-latest' }}
+        run: |
+          cp -r focus_converter_base/* ./
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+      - name: Install poetry
+        uses: abatilo/actions-poetry@v2
+      - name: Setup a local virtual environment
+        run: |
+          poetry config virtualenvs.create true --local
+          poetry config virtualenvs.in-project true --local
+          poetry lock
+      - uses: actions/cache@v3
+        name: Define a cache for the virtual environment based on the dependencies lock file
+        with:
+          path: ./.venv
+          key: venv-${{ hashFiles('poetry.lock') }}
+      - name: Install dependencies
+        run: |
+          poetry install
+      - name: isort
+        run: |
+          poetry run isort focus_converter/ -c
+      - name: black
+        run: |
+          poetry run black focus_converter/ --check
+      - name: flake8
+        run: |
+          poetry run flake8 focus_converter/ --select=-E,W,F,N,I
diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml
@@ -0,0 +1,34 @@
+name: Publish
+on:
+  create:
+    tags:
+      - 'v\d\.\d\.\d'
+      - 'v\d\.\d\.\d-(dev|rc)\d'
+
+jobs:
+  publish:
+    permissions:
+      id-token: write
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Copy converter_base to current directory Windows
+        if: ${{ matrix.os == 'windows-latest' }}
+        run: |
+          xcopy /E /I /Y focus_converter_base\* .\
+      - name: Copy converter_base to current directory Linux, MacOS
+        if: ${{ matrix.os != 'windows-latest' }}
+        run: |
+          cp -r focus_converter_base/* ./
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.9
+      - name: Install poetry
+        uses: abatilo/actions-poetry@v2
+      - name: Install dependencies
+        run: |
+          find -type l -exec bash -c 'ln -f "$(readlink -m "$0")" "$0"' {} \;
+          poetry build
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1