A company that ships every week rebuilds its own site faster than search engines can keep up with it. Pages appear, get renamed, get merged into something else, and disappear again — all before anything has been recorded. The result is a site that is technically live and, from the outside, only partially visible.
Teams working at this pace tend to treat discovery as an afterthought, and the assumption underneath is reasonable enough: publish it and it will be found. That holds for a site that changes twice a year. It stops holding for a site where a documentation generator emits four hundred pages on Tuesday and a routing change renames half of them on Friday. At that point discovery becomes a thing you manage, not a thing that happens.
The gap between shipped and recorded
The first sign is rarely a drop in visits. It is a mismatch: the team knows a page exists, and no query in the report ever returns it. Nobody investigates, because there is no alert for a page that was never picked up. It simply is not there, and absence generates no signal.
| What the team believes | What is actually true | How long it goes unnoticed |
|---|---|---|
| The docs are published | Recorded up to two levels deep, no further | Until someone searches for a deep page |
| The old URLs redirect | They redirect through three hops | Until a slow crawl is investigated |
| The blog is indexed | Half of it is, arbitrarily | Usually a full quarter |
| The app pages count | They render after load and are seen empty | Indefinitely |
The fourth row is the one that costs the most in a company whose product is a web application. If the meaningful content of a page is assembled in the browser after delivery, what gets recorded is whatever was in the source — often a shell with a loading state. The page exists for users and does not exist for anything else, and every hour spent analysing reports about it is wasted, because the underlying record was never created.
There is a second-order effect worth naming. A team that has been burned by this once tends to overcorrect, submitting everything continuously and treating the daily allowance as something to exhaust rather than to spend. That produces a different failure with the same appearance: capacity permanently occupied by pages that were already fine, and nothing available when a genuine emergency arrives. Both failures look identical from the outside — pages taking a long time to appear — and they call for opposite responses.
Two routes in, and only one of them you control
Links from pages already known
The default mechanism. Anything reachable in one or two steps from a well-known page tends to get picked up; anything five steps down often does not.
- Depends entirely on internal structure
- Punishes deep hierarchies quietly
Explicit submission
You hand over a list. Useful precisely when the structure is deep, new, or has just changed under everyone's feet.
- Works regardless of depth
- Bound by hard daily limits
For a fast-moving product site the second route is not optional. A generated documentation tree is deep by nature — sections inside sections inside versions — and a structure that reaches five levels will have its deepest pages discovered late or not at all. Submitting explicitly bypasses the depth problem entirely, which is why it matters more here than for an ordinary marketing site.
Route one is not something to abandon, though. Improving internal structure has a permanent effect where submission has a temporary one: a docs tree with cross-links between related sections, rather than only parent-to-child links, is easier to traverse forever afterwards, whereas a submitted address benefits only from that submission. The right posture is to fix the structure so route one works and use route two for the moments when structure cannot help — a rename, a migration, a version bump that changes several hundred addresses at once.
The submission limits, stated plainly
What fits in a day and in a job
These are the numbers that decide whether a relaunch takes two weeks or two months.
- One thousand addresses per day, counted per account. Not per domain. A company with a product site, a docs site and a careers site shares one daily allowance across all three.
- Ten thousand addresses in a single job. The job accepts the full list at once and works through it against the daily ceiling, so a ten-thousand-page submission is a ten-day operation by arithmetic.
- Two jobs running, twenty waiting. Order matters. A large low-priority job submitted first will occupy a slot while the urgent one waits behind it.
- Sitemaps nested up to three levels, one thousand per job. Enough for a generated docs tree, provided the index files are actually linked rather than merely present on disk.
What those numbers mean for a release
The planning error is almost always the same one.
- Assuming the allowance multiplies by domain. A team with three properties plans against three thousand a day, gets one thousand, and concludes the system is slow rather than that the plan was wrong.
- Submitting everything instead of what changed. A full sitemap resubmission spends the allowance on pages that were already fine. The changed pages then wait behind them.
- Ignoring the queue depth. Twenty waiting jobs sounds generous until a nightly automation files one per build and fills the queue with duplicates.
- Submitting before the redirects are clean. Every address that resolves through a chain consumes allowance and delivers less than a direct one.
None of these limits are obstacles in themselves. They become obstacles when a plan is built on the assumption that capacity is unlimited, which is the default assumption when nobody has read the numbers. Ten minutes spent dividing page count by daily allowance produces a realistic schedule and removes the most common cause of a relaunch that appears to stall for no reason.
It is also worth being clear about what these limits are not. They are not a quality signal, and staying under them earns nothing. They are a scheduling constraint, in the same category as a build queue or a deployment window, and the correct response is the same one an engineering team would apply to any other queue: know the throughput, order the work by importance, and do not enqueue things that do not need to run.
Spending the allowance on the right thousand
| Situation | What to submit first | Why |
|---|---|---|
| Routing change on the product site | Changed addresses only | Unchanged pages need no allowance |
| New documentation version | Index pages, then leaves | Index pages create the second route in |
| Whole site moving | Highest-traffic pages first | Recovers the majority of impact early |
| Routine week | Nothing at all | Unspent allowance is not wasted |
The last row is the least intuitive and the most useful. A weekly automated submission of every address on the site accomplishes nothing except keeping the queue permanently full, so that when something genuinely urgent happens there is no free capacity for it. Submitting on change rather than on schedule keeps the allowance available for the moment it is actually needed.
Chains that quietly halve your capacity
A site that has been through two rebuilds accumulates redirect chains without anyone deciding to create them. The first rebuild pointed the old path at a new one; the second pointed that new one somewhere else again. Each address now resolves through two or three hops, and every hop is work that produces nothing.
- Collapse every chain to one hop. Point the original address directly at the final destination. This is mechanical work and usually a single afternoon.
- Do it before submitting, not after. Submitting a chained address spends allowance to discover a detour.
- Watch for chains that cross domains. Product site to docs site to marketing site is common after a reorganisation and is the slowest variety.
- Delete redirects that lead nowhere useful. An address pointing at a page that no longer serves its original purpose is better retired than preserved out of caution.
The scale of this problem is easy to underestimate. A site with eight hundred legacy addresses, each resolving through two hops, is asking for roughly two thousand four hundred requests where eight hundred would do — three times the necessary work, spread across an allowance that was already the binding constraint. Cleaning that up costs nothing and returns capacity immediately. Keeping the before-and-after state visible in one workspace is what makes the improvement provable rather than assumed.
One further note on chains: they are worth auditing after every reorganisation, not once and forever. Each rebuild adds a layer, and a layer added on top of an existing chain lengthens it rather than replacing it. Building the audit into the release checklist — a single automated check that no address resolves through more than one hop — costs an hour to set up and prevents the problem from ever accumulating again.
Confirming the record actually changed
Submission is a request, not a guarantee. The only meaningful confirmation is that the address subsequently appears in reports as having produced impressions, and that takes time. Between the two sits a period where nothing is knowable, and this is precisely where teams tend to resubmit repeatedly out of impatience — spending allowance to no effect.
A workable discipline: submit once, wait a full week, then check coverage before doing anything else. If an address has not been picked up after two weeks despite a clean submission, the cause is almost always on the page itself — thin content, a delivery problem, or a duplicate elsewhere on the site — and resubmitting a third time will not address any of those. Watching the sequence in a connected environment makes the difference between the two cases visible without a separate spreadsheet.
The impatience is understandable, because everything else in a fast-shipping company gives feedback within minutes. This does not. Building the wait into the process explicitly — a calendar entry a week out, rather than a vague intention to check later — is what stops the resubmission reflex, and it also produces a record of when each submission was made, which turns out to matter when someone asks three months later why a particular page took so long.
When the generator produces more than it should
Documentation tooling and content management systems both tend to emit more addresses than a human would. Tag pages, filter combinations, per-version copies of identical text, printable variants — each is a legitimate address and most of them carry no independent value. Submitted wholesale they consume the daily allowance and dilute the site with near-identical pages.
Pages a person would search for
The current version of each document, the landing pages, anything with text that exists nowhere else on the site.
- Submit deliberately
- Link from an index page
Machine-generated variants
Old versions, tag listings, filter permutations, print views. Real addresses with no independent reason to be found.
- Never submit
- Keep out of the sitemap
Keeping the kept-and-excluded decision written down somewhere the whole team can see it — rather than encoded only in a build config — prevents the next person from quietly re-enabling a category because it looked like an omission. A shared view in the same environment that holds the coverage data serves that purpose well enough, since the decision and its consequences then sit next to each other.
Deciding this once, at the generator level, is far cheaper than deciding it repeatedly per release. A configuration that emits only the addresses worth finding turns an unbounded problem into a bounded one, and the daily allowance stops being a constraint the moment the list it applies to is the right length.
Version history is the hardest case, because there is a legitimate reason to keep older documentation available and a legitimate reason not to have five near-identical copies of the same page competing with each other. The usual resolution is to keep old versions reachable for users while excluding them from submission and from the sitemap — available to anyone who navigates there, invisible to discovery. That satisfies both requirements without compromise.
The order of operations
Sequence matters here more than in most work, because each step invalidates measurements taken before it. Fixing delivery after analysing reports means the analysis was performed on incomplete data; cleaning redirects after submitting means the submission was partly wasted.
| Step | Task | Effort |
|---|---|---|
| One | Confirm content arrives in the delivered source | A morning |
| Two | Collapse redirect chains to a single hop | An afternoon |
| Three | Trim the generator's output to pages worth finding | A day, once |
| Four | Submit changed addresses, priority first | Minutes per release |
Step one is settled by a technical review, and it is the only step that cannot be deferred — everything downstream depends on it. Which pages are worth keeping in step three is partly a demand question, answerable through keyword research rather than by intuition about the documentation tree. Where the same text appears at several addresses, the fix belongs to on-page work rather than to submission, because no amount of submitting resolves a duplicate. Running all three views from a single account keeps the sequence honest, since each step's result is visible to the next.
Check what has actually been recorded
Questions that come up during a release
Is the daily limit per domain or per account?
Per account. One thousand addresses a day covers everything in it. A team with three properties has to divide, not multiply — and the division should be a deliberate decision, otherwise whichever job was filed first takes the capacity regardless of importance.
Should we resubmit after every deploy?
Only for addresses whose content actually changed. A blanket resubmission on every build fills the queue with duplicates and leaves nothing free when something urgent happens. Wire the automation to the set of changed files, not to the deploy event.
Our documentation is five levels deep. Is that a problem?
For the ordinary route in, yes — depth reduces the chance that the deepest pages are found at all. Explicit submission works around it, and so does adding index pages that link across the tree rather than only down it. Doing both is normal for a generated docs site.
How many hops is a redirect chain allowed to have?
Aim for one. Two is tolerable and wasteful; three or more is where the cost becomes visible in how long a large submission takes to work through. Cross-domain chains are the worst case and the most common after a reorganisation.
Why is a page still missing two weeks after submission?
At that point the submission is not the issue. Look at the page: whether its content is present in the delivered source, whether the same text appears elsewhere on the site, and whether anything on the site links to it. Resubmitting a third time addresses none of those.
Can we submit ten thousand addresses in one go?
Yes, a single job accepts that many. It will then work through them at the daily rate, so a full ten thousand is roughly ten days of processing. Plan the ordering accordingly — the pages that matter most should be at the front, not distributed evenly through the list.