Member of Technical Staff (Web Access Engineer, Crawler)
Could this role work from Kenya?
Kenya eligibility needs confirmation
- Stated location
- London
- Listing updated on our site
- 14 Nov 2026, 00:01 EAT
Last seen in the live employer feed: 14 Nov 2026, 00:00 EAT. This confirms a feed listing, not employer endorsement of an application.
- No clear location or timezone eligibility signal was found in this listing.
These are automated wording checks, not confirmation from the employer. The update date records a site update, not a hiring check. Review location, working hours, experience, and current availability in the application portal. Employer identity remains hidden here.
Your application workflow
- 1. Check requirements
- 2. Tailor your CV below
- 3. Apply through the portal
- 4. Track your progress
AI Application Copilot
Prepare role-focused CV bullets and a cover letter to review
| Employer | Verified Direct Employer (Protected) |
|---|---|
| Location | London |
| Remote type | Remote |
| Posted | 20h ago |
| Application Status | Pro Membership & Free Pass Access |
About this role (Direct Employer Details Protected)
[Employer] is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.
Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.
Responsibilities
- Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
- Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
- Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
- Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
- Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
- Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.
Qualifications
- 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.
- Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.
- Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.
- Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.
- Comfort working independently when the path forward is not fully defined.
Nice to have
- Experience developing CDN, caching, or web performance systems.
- Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.
- Contributions to a browser engine or JavaScript runtime.
Unlock 0-Second Direct Apply
This role was published 20h ago. Global recruiters typically freeze applications after receiving the first 50 CVs. Pro members and Free Pass holders reveal the employer identity and get immediate 0-second direct apply access before positions fill.
Top of Recruiter Inbox
First 20 applicants receive 80% of recruiter interviews.
AI ATS Match Tailoring
Instant role-targeted bullets & cover letters.
Remote Contractor Toolkit
USD invoices, W-8BEN guide & CRM pipeline.
Instant M-Pesa STK push · 7-Day Money-Back Guarantee · Be in the first 20 applicants
Related Remote Jobs Open to Kenya
Explore similar international remote opportunities
Member of Technical Staff (Software Engineer, Fraud Prevention)
Verified Direct Employer · Vetted
About the role [Employer]'s fraud prevention team protects our products, users, and infrastructure from fraud and abuse across the whole lifecycle: s…
Member of Technical Staff (Search Crawler Analyst)
Verified Direct Employer · Vetted
The internet is vast, containing trillions of URLs. [Employer]’s crawling and storage system is complex and has multiple stages (URL discovery, crawli…
Member of Technical Staff (Machine Learning Engineer, Search Agent RL)
Verified Direct Employer · Vetted
[Employer] is seeking an experienced Machine Learning Engineer to advance how AI systems search, reason, and work together to solve complex problems.…