When a CUDA Call Returns — and When the Work Is Actually Done

A GPU kernel launch is not a function call that computes something. It is a request placed on a queue.

launch kernel   → returns in ~10 µs      (host time to enqueue)
kernel runs     → 50 ms                  (device time, later)

Measured: enqueuing a kernel cost 9.7 µs of host time while the kernel itself ran for 50 ms. Kernel launches and most asynchronous CUDA calls behave this way, and a whole family of subtle pipeline bugs is a disagreement about when the work is done.

The first consequence is that your results may not be there. GPU work queued on one stream is ordered automatically — the danger is at the boundary where the host reads, or where two streams meet with no dependency between them. The common shape: results are copied asynchronously into a pinned host buffer and something reads that buffer before the copy lands.

That is measured here. A search enqueued 256 queries and returned in 0.08 ms; the answers were ready at 2.13 ms. Read in between, every one of the 256 rows was the previous batch's answer — not garbage, not zeros, a complete and plausible result with a recall of 0.1% instead of 100%. Nothing errors. This is the failure mode to suspect whenever a GPU pipeline is "almost right".

The second is that traditional allocation hides a synchronization point. Called in the middle of that 50 ms kernel, a plain cudaFree blocked for 50.2 ms waiting for the outstanding work to finish. The stream-ordered pair (cudaMallocAsync / cudaFreeAsync) does not wait that way — the free returned in 27 µs and is ordered against the work already queued on the stream instead, which also means the lifetimes and stream dependencies are now yours to get right. Allocating per iteration with the ordinary API turns an async pipeline into a synchronous one without a single visible synchronize().

The third is that "async" describes the API, not the transfer. From ordinary pageable host memory the driver may have to stage the data through a pinned buffer of its own, and the call blocks while it does: 42.9 ms here, against 9.6 µs from a page-locked buffer. Pinning is what makes the asynchrony reliable — though it only permits overlap, it does not create it: that still needs independent streams and no dependency in the way. If a profile shows a memcpy blocking the host, the allocation on the host side is usually the reason.

A stream is the ordering primitive. Work on the same stream runs in issue order; independent streams may overlap when the hardware has room. The exception worth knowing is the legacy default stream, which synchronizes against other blocking streams — so which stream you use matters as much as the order you use it in. All of which is why a serious GPU function signature carries a stream parameter: the caller decides when, and relative to what.

search_async(..., cudaStream_t stream)   // who owns it, where it lives, WHEN it runs

The practical habits that follow:

• Synchronize (or wait on an event) before reading results on the host — and put that boundary somewhere obvious, not inside a helper.
• Pin the host buffers whose transfers you need to overlap; pinning everything is not the goal, and pinning is slow in itself — 64 MiB took 33 ms to allocate.
• Keep allocation out of the steady-state loop, or use the stream-ordered allocator.
• When timing with a host clock, synchronize after submitting the work as well as before — otherwise you are timing the enqueue, the 9.7 µs number rather than the 50 ms one. For device time, record CUDA events in the stream being measured.

This is also why frameworks reach for CUDA graphs and for kernel fusion, which fix different things: a graph replays a whole sequence of launches instead of issuing them one at a time, while fusion removes launches and the intermediate reads and writes between them. At small batch sizes both stop being micro-optimizations.

Measured, with the traces and a library API built around the "when" question, in From Python Call to GPU Kernel.