ScrapeMaster 3.0 - Installation Guide
Discover the project in action
Watch Video1️⃣ Create a Virtual Environment
Run the following command to create and activate a virtual environment:
bash
python -m venv venv
2️⃣ Install Dependencies
Create a requirements.txt file and copy the following dependencies:
plaintext
openaipython-dotenvpandaspydanticrequestsbeautifulsoup4html2texttiktokenseleniumreadability-lxmlstreamlitstreamlit-tagsopenpyxl
Then install all dependencies with:
bash
pip install -r requirements.txt
3️⃣ Add Your API Key
Create a .env file and add your OpenAI API key:
plaintext
OPENAI_API_KEY=sk-xxxxxxxx(place your own key)GOOGLE_API_KEY=AIzaSyxxxxxxxGROQ_API_KEY=gskxxxxxxxxx
4️⃣ Download ChromeDriver
Download ChromeDriver from the official website:Chrome for Testing availability
5️⃣ Create the Scraper Script
Save the following script as scraper.py:
python
// Loading...
6️⃣ Create the Streamlit App
Save the following as streamlit_app.py:
python
// Loading...
7️⃣ Create the assets.py
Save the following as assets.py:
python
// Loading...
8️⃣ Create the pagination_detector.py
Save the following as pagination_detector.py:
python
// Loading...
9️⃣ Run the Streamlit App
Run the following command:
bash
streamlit run streamlit_app.py