Skip to content

Script 🐍 that pass through web page like spyder 🪲 and get valuable link for you 💰

License

Notifications You must be signed in to change notification settings

harshareddy794/Web-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Web Crawler

forthebadge made-with-python ForTheBadge built-with-love ForTheBadge uses-badges Open Source Love svg1 GitHub license

Introduction

Being Cyber geek 💻 its always fun to test web applications. Which information gathering is one of the biggest part of the testing. Some times it is very important to know all links in a website, which will be very useful in testing any web application. So the only way to retrive all links in a web site is using crawling 🪲 a web application.

Crawler video

Libraries Used

  • Beauty soup - To retrive and process html data
  • urllib - To send request to web server
  • XLWT - to save retrived data to excel sheet
  • Date time - To get current date and time

Setup

Initially we have to install all required libraries Install all require libraries using command

pip install -r"requirements.txt"

Usage

Run script using python command in your command prompt/terminal

python Crawler.py

Disclaimer ‼️

Crawling of web pages is restricted ⛔ in some sites without written proof 📝 from the owner of site. I/Myteam does not took the responsiblity of wrong usage of this tool. Use at user own risk 💯

contribution

Every one who love open source projects and web developement can contribute to my project ❤️ .Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated. Every contribution is valuable and if you feel any issues in the project feel free to open an issue ✊

Contact

Email: [email protected]

LinkedIn: www.linkedin.com/in/harshareddy794

Acknowledgement

Thank you every who helped me in building such an amazing project

License

MIT

About

Script 🐍 that pass through web page like spyder 🪲 and get valuable link for you 💰

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages