Extract

Static Data

It is possible to create static datasets by hardcoding data into the code and returning a LazyFrame

import polars as pl

def transform():    
    return pl.LazyFrame({
        "country_name": ["Switzerland", "Germany", "France", "Italy", "Spain"],
        "iso_code": ["CH", "DE", "FR", "IT", "ES"],
        "population": [8600000, 83000000, 67000000, 60000000, 47000000]
    })

Or by creating a CSV file in the repository and loading it via pl.scan_csv("./static.csv")

Extracting from other Systems

It is possible to extract data from various sources like databases, file shares, web crawling, or external APIs, either through ready-made connectors or by writing custom code that connects to a custom API.