Useful Cheatsheets
Build / AI /

Parallel Tool Calling

Execute multiple concurrent tool requests simultaneously using Promise.allSettled and rate-limited batch workers.

TL;DR

  1. Execute multi-tool responses concurrently using Promise.allSettled execution workers.
  2. Throttle maximum simultaneous tool operations using the p-limit library.
  3. Return individual response blocks mapped to each corresponding tool_call_id.
Jump to section

Concurrent Dispatch Architecture

    AllSettled Worker

    Execute multiple tool calls in parallel with isolated error safety.

    const results = await Promise.allSettled(
  toolCalls.map(async call => ({
    id: call.id,
    data: await executeTool(call.name, call.args),
  }))
);
    Result Block Formatter

    Map settled promise states into appropriate LLM tool response messages.

    const msgs = results.map((r, i) => ({
  role: 'tool',
  tool_call_id: toolCalls[i].id,
  content: r.status === 'fulfilled'
    ? JSON.stringify(r.value.data)
    : JSON.stringify({ error: (r as any).reason }),
}));
    Concurrency Throttling

    Constrain maximum concurrent tool requests using p-limit.

    import pLimit from 'p-limit';
const limit = pLimit(3); // Max 3 concurrent calls
const tasks = toolCalls
  .map(c => limit(() => executeTool(c)));
const out = await Promise.allSettled(tasks);

Partial Failure Recovery

    Granular Error Isolation

    Segregate successful tool data from failed execution exceptions.

    const payload = results.map(res => {
  if (res.status === 'fulfilled') {
    return { success: true, data: res.value };
  }
  return { success: false, error: 'Unavailable' };
});
    Model Guidance Feedback

    Prompt model to synthesize partial answers from successful calls.

    const feedback = {
  role: 'tool',
  tool_call_id: failedCall.id,
  content: 'DB timeout. Use available results.',
};
    Deadlock Prevention Check

    Verify that total tool responses exactly matches request count.

    if (toolResponses.length !== toolCalls.length) {
  throw new Error('Mismatched tool call IDs detected');
}

Cross-Provider Concurrency

    OpenAI Multi-Call

    Extract and execute multiple tool_calls from chat completion.

    const calls = res.choices[0].message.tool_calls ?? [];
const toolOutputs = await Promise.all(
  calls.map(c => dispatchOpenAiTool(c))
);
    Claude Multi-Tool Content

    Extract multiple tool_use blocks from Anthropic message content.

    const toolBlocks = res.content.filter(
  b => b.type === 'tool_use'
);
const responses = await Promise.all(
  toolBlocks.map(b => dispatchClaudeTool(b))
);
    Batch Turn Resumption

    Send all parallel tool results back in a single follow-up message.

    await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20241022',
  max_tokens: 1024,
  messages: [...history, assistantMsg, {
    role: 'user',
    content: responses,
  }],
});

Performance Optimization

    Batch Latency Logging

    Measure overall wall-clock time saved through parallel execution.

    const t0 = performance.now();
await Promise.all(toolTasks);
const duration = performance.now() - t0;
metrics.timing('parallel_tool_duration_ms', duration);
    Deduplicated Tool Calls

    Filter identical tool arguments before dispatch to avoid waste.

    const entries = calls
  .map(c => [JSON.stringify(c.args), c]);
const unique = Array.from(new Map(entries).values());
    Sub-Request Cache

    Serve repeated parallel tool queries from in-memory cache.

    async function cachedExecute(call: ToolCall) {
  const a = JSON.stringify(call.args);
  const key = `tool:${call.name}:${a}`;
  return cache.getOrSet(key, () => execute(call), 60);
}

Tips

  1. Use Promise.allSettled instead of Promise.all so that a single failed tool call does not abort other successful operations.
  2. Include partial failure descriptions in error result blocks so models adapt using successful tool-results.

Warnings

  1. Limit parallel tool execution concurrency using p-limit to prevent exhausting downstream database connection pools and API quotas.
  2. Ensure every single tool_call_id receives a response block even if an operation was rejected or skipped to prevent conversation deadlocks.

In Practice

FAQ