Robots.txt File SEO Me Kya Hota Hai?

Robots.txt File SEO Me Kya Hota Hai?

Robots.txt file

Robots.txt File Kya Hai?

Robots.txt ek text file hoti hai jo website ke root directory me store hoti hai. Ye search engine crawlers (bots) ko batati hai ki website ke kaunse pages ya sections crawl karne hain aur kaunse nahi.

Robots.txt SEO Me Kyon Zaroori Hai?

  1. Crawling Control
    – Aap search engine bots ko guide kar sakte hain ki kaunse pages index hone chahiye aur kaunse nahi.

  2. Duplicate Content
    Duplicate Content Se Bachav
    – Agar ek website me duplicate content hai, to usse avoid karne ke liye robots.txt ka use hota hai.

  3. Server Load Kam Karna

    Server Load

     – Aap un pages ko block kar sakte hain jo crawl karne ki zaroorat nahi hai, jisse server pe load kam hota hai.

  4. Sensitive Pages Ko Hide Karna

    Sensitive pages

    – Aap confidential ya private pages ko search engine me show hone se rok sakte hain.




Example: Robots.txt File Ke Practical Examples



1. Sabhi Crawlers Ko Puri Website Crawl Karne Ki Permission Dena

txt
User-agent: *
Disallow:

Explanation:

  • User-agent: * → Sabhi search engine bots ke liye apply hoga.

  • Disallow: → Kuch bhi block nahi kiya, yani poori website accessible hai.

2. Sabhi Crawlers Ko Puri Website Crawl Karne Se Rokna

txt
User-agent: *
Disallow: /

Explanation:

  • / ka matlab hai poori website ko block karna.

3. Ek Particular Page Ko Block Karna

txt
User-agent: *
Disallow: /private-page.html

Explanation:

  • /private-page.html ka matlab hai ki sirf ye page search engines me index nahi hoga.

4. Ek Folder Ko Block Karna

txt
User-agent: *
Disallow: /admin/

Explanation:

  • /admin/ folder aur iske andar jitne bhi files hain, wo crawl nahi hongi.

5. Sirf Google Bot Ko Block Karna

txt
User-agent: Googlebot
Disallow: /

Explanation:

  • Ye sirf Google ke bot ko website access karne se roke ga.

6. Sitemap Ka URL Dena (Best SEO Practice)

txt
User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Explanation:

  • Sitemap: search engines ko batata hai ki website ka XML Sitemap kaha hai, jo indexing ke liye helpful hota hai.


Comments

Post a Comment

Popular posts from this blog

Digital Marketing kya hai aur yeh kaise kaam karti hai?

Image
Digital Marketing Kya Hai?  Digital Marketing ek tarika hai jisme hum internet aur digital platforms ka use karke apne products ya services ko promote karte hain. Ye online marketing ka ek hissa hai jo businesses ko unke target customers tak pahunchne aur sales badhane me madad karta hai. Digital Marketing ke Pramukh Types   Search Engine Optimization (SEO Apni website ko Google jaise search engines me rank karwane ka process. Keywords, backlinks aur content optimization ka use hota hai. Example: Agar aap ek bakery chalate hain, to "best bakery in Delhi" jaisa keyword optimize karke Google pe top pe laa sakte hain. 2. Social Media Marketing    Facebook, Instagram, Twitter, LinkedIn jaise platforms par branding aur promotion.  Paid ads aur organic content ka mix use hota hai.   Example: Instagram pe ek clothing brand apne naye designs ka promotion kar raha hai. 3. Content Marketing  Blogs, videos, infographics, aur articles ke through valuable cont...
Read more

URL kise kahte hain?

Image
URL Kya Hai? 📌 URL (Uniform Resource Locator) ek web address hota hai jo kisi webpage , file, ya online resource ka location batata hai. Jab aap Google Chrome, Firefox, ya kisi aur browser me koi website open karte hain, to aap URL ka use kar rahe hote hain. Example: ✅ https://www.google.com ✅ https://www.youtube.com/watch?v=abcd1234 URL Ke Parts (Components of URL) Ek URL ke kai parts hote hain jo uski structure aur function ko define karte hain: 🔹 1. Protocol: ( https:// ya http:// ) Ye batata hai ki browser aur server ke beech data kaise transfer hoga. HTTPS (Secure) aur HTTP (Non-Secure) protocols hote hain. 🔹 2. Domain Name : ( www.google.com ) Ye batata hai ki kaunsi website ko access karna hai. Example: www.facebook.com , www.amazon.in . 🔹 3. Path: ( /search , /about ) Ye batata hai ki website ke kis specific page ko open karna hai. Example: https://www.example.com/about ( /about ek webpage hai). 🔹 4. Query String (Optional): ( ?q=search+term ) Ye additiona...
Read more

Website ke liye Domain Name kya hota hai aur kyun zaroori hai?

Image
  Domain Name Kya Hai? 📌 Domain Name ek unique web address hai jo kisi website ko internet par identify karta hai. Ye human-friendly name hota hai jo kisi IP Address (e.g., 172.217.160.78 ) ko represent karta hai, taki log asani se websites access kar sakein . Example: ✅ www.google.com ✅ www.facebook.com ✅ www.amazon.in Agar domain name na ho , to hume websites open karne ke liye complex numbers (IP Address) yaad rakhne padenge! Domain Name Ke Parts (Components of a Domain Name) Ek domain name ke 3 main parts hote hain: 1️⃣ Subdomain: ( www. ya koi aur custom name) Example: www.example.com me www ek subdomain hai. blog.example.com me blog ek subdomain hai. 2️⃣ Second-Level Domain (SLD): ( example ) Ye website ka actual name hota hai. Example: google.com me google SLD hai. 3️⃣ Top-Level Domain (TLD): ( .com , .org , .in ) Ye domain ka category ya country batata hai. Example: .com → Commercial websites .org → Organizations .in → India-based websites Domain Name...
Read more