Simple data pipeline concept 1-1-1 Rule
Nasdaq ☆ 100 1 ☆ 1 ☆ 1 Rule Data Pipeline (Starter-kit)
Code with step-by-step description:
https://github.com/DataSpaceCafe/1-1-1-rule-data-pipeline-stock-nasdaq-100
Today I want to share a simple usecase that helps in the idea of doing 1 ☆ 1 ☆ 1 Rule Data Pipeline by writing code. Understand the principle ETL data pipeline process simple.
"Retrieve stock data － evaluate value ☆ upload Drive Use Create function on google app script Timer (trigger) Analyze data with Gemini AI Model" fully automated.
Simple step-by-step work:
1) Extract - Pull stock list
- Auto-pull the Nasdaq ☆ 100 list from Wikipedia.
- If you can't pull it, you will fallback the file 'nasdaq100 _ tickers.csv'.
2) Transform - Valuation
- Retrieve basic information from yfinance: price, EPS, PE, growth, book value.
- Do data cleansing (format ticker, delete repeat value, convert data type)
- Calculate: PEG Ratio, Graham Value, Margin of Safety
- Summarized as' valuation 'and' valuation _ hunter 'results.
3) Load - Save and Upload
- Save as CSV in 'data /'.
- Upload up Google Drive (OAuth / Service Account)
- If you open 'GOOGLE _ DRIVE _ CONVERT _ TO _ SHEETS = true', it will be Google Sheets.
4) Analytics - View results in Looker Studio
- Create a Data Source from a file located on the Drive.
- Set the refresh after the market closes.
- Make an undervalued / overvalued / fair break graph immediately.
5) AI Summary - Auto Analysis with Gemini
- Apps Script will retrieve the latest files from Drive.
- Send it to Gemini － 2.5 ☆ flash analysis.
- Send summary in daily email
6) Monitoring - There is a log every time.
- Create a log file by date.
- Performance log per function (time, CPU, Memory)
See for example the code in the first comment. 😁