Crawling smarter, not harder
Vložit
- čas přidán 10. 09. 2024
- In this episode of SOTR, John Mueller, Lizzi Sassman, and Gary Illyes talk about misconceptions around crawl frequency and site quality, what's challenging about crawling the web nowadays, and how search engines could crawl more efficiently.
Resources:
Episode transcript → goo.gle/sotr07...
Gary's post on LinkedIn → goo.gle/3YAT55q
Crawling episode with Dave Smart → goo.gle/3WShUsf
If-Modified-Since → goo.gle/3ywXvja
About the IETF → goo.gle/3SGVVlo
Robots Exclusion Protocol → goo.gle/4dgmBSg
Proposal for new kind of chunked transfer → goo.gle/3AgMF1c
Listen to more Search Off the Record → goo.gle/sotr-yt
Subscribe to Google Search Channel → goo.gle/Search...
Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
#SOTRpodcast
00:13 Discussion on crawling updates and strategies.
02:53 Google's crawling frequency does not necessarily indicate site quality
07:54 Site owners may want increased crawling over time.
10:19 Optimizing crawling process by understanding server limitations and content quality
15:32 Sending back only headers in HTTP response makes it more efficient
17:38 Implementing compartmentalized changes can save resources.
22:05 Optimizations for crawling include URL parameter handling
24:22 URL parameters impact crawling efficiency
28:42 Challenges of smart crawling
30:53 Crawling efficiently depends on site size and resource awareness
34:53 Hosting companies should assist customers in troubleshooting website crawling issues
37:16 Client-side connection problem solutions
Regarding parameters, the former URL parameters tool in GSC seemed to handle this situation. When that was taken away, we started seeing TONS of parameterized URLs showing up on SERPs and in our analytics from organic search (including URLs with advertising attribution parameters, which is especially fun). Now it's a problem????? When it went away Google told us they were able to figure out which parameters they could ignore, but....evidently not. Maybe just bring that tool back and let us take care of it ourselves? It was working.
00:13 Discussion on crawling strategy and updates
02:53 More crawling does not necessarily mean a better site.
07:54 Website owners may request for increased crawling over time, but it's not always feasible.
10:19 Optimizing crawling efficiency and server impact
15:32 Sending smaller 304 response for efficiency
17:38 Crawling smarter can save resources for the internet.
22:05 Crawling optimizations discussion
24:22 URL parameters can result in infinite versions of a page
28:42 Challenges in optimizing crawling methods
30:53 Importance of awareness and resources in crawling efficiently
34:53 Hosting companies should assist customers better in troubleshooting crawling issues.
37:16 Improving client-server communication and messaging strategies.
Thank you too
You need to crawl before you can walk the walk!
This is the funniest episode, lmao
John's on point with the jokes
So how Google bots honour sitemaps that contain the date of last modification in .... tags?
A lot of sitemaps get these wrong (many always use "now" as the last change date), so it's a bit hit & miss. If you can do it right, then do it. If you can't set it properly, then it's better to submit the sitemap without dates (which is also fine).
Who determines the quality of content?
I love the personality switch between Gary and Lizzy. :-)
32:54 32:54
Почему мировой гигант проводит радио передачи? К тому же с неприятным юмором
likability aside, what's the contradiction there I don't get it
Hello pin comment please 😢
Why?
📌