Ingest

Data Space does not provide a mechanism to upload data. Therefore, the ingest needs to come from an external URL such as, for example, Google Drive.

Google Drive

To ingest a CSV file from Google Drive, you can first enable sharing on the file by following the instructions on the Google Drive Help Page.

The generated share link will look something like this: https://drive.google.com/file/d/1Se7_LKZykBWweXpBths1oCmgGTGK4yyD/view?usp=sharing

This link is meant to open the Google Drive web interface. However, since we want the file itself, we have to modify the link. The file ID needs to be extracted from the original URL and combined with the direct file access link:

https://drive.google.com/uc?id=1Se7_LKZykBWweXpBths1oCmgGTGK4yyD

Following is the full code

import polars as pl
import os

url = 'https://drive.google.com/uc?id={os.environ['CSV_FILE_ID']}'

def transform():    
    df = pl.read_csv(url)
    return df

Typically a file URL will contain an access token. It is advised not to hard code the key into the string but rather use the secrets store to inject a key during the build.