Skip to content
Trending storyDeveloping

Together AI's GPU clusters auto-repair faulty nodes, per company blog and post

2 articles2 sourcessince Oct 26Last article

Overview

AISummary of 2 articles

Together AI says its auto repair system detects GPU cluster faults such as Xid errors, ECC errors and InfiniBand link flapping, and returns faulty nodes to health.

Detection runs inside the virtual machine and on the host, and each fault is matched to the lightest repair action. When a fault recurs, repairs escalate from reboot to reprovision to migration, and migrated machines are re-validated before returning to the pool.

Customers choose whether repairs wait for approval or run automatically, down to individual XID codes, according to the company's blog and a follow-up post on X. The blog is first-party and includes no independent measurements of repair outcomes.

Written by AI from the articles below · updated Oct 27, 6:30 PM ET

Check the sources:

Developments

2 developments

  1. Oct 27, 6:26 PM ET · 1 article
    Together AI adds automatic repair for GPU clusters that recovers nodes from faults
    Together AI: Together GPU clusters now auto-repair faulty nodes back to healthy
  2. Oct 26, 8:00 PM ET · 1 article
    Together AI explains how auto repair turns GPU faults into healthy nodes
    Together AI Blog: Together AI explains how auto repair turns GPU faults into healthy nodes

Article timeline

The articles in this story. Times are ET.

Oct 27
  1. Together AIOfficial
    Together GPU clusters now auto-repair faulty nodes back to healthy

    AITogether AI says its GPU clusters can now automatically return a faulty node to health, with health checks identifying the failure and the platform choosing the lightest repair. Users decide which actions can run unattended, down to individual XID codes.

Oct 26
  1. Together AI BlogOfficialPick
    Together AI explains how auto repair turns GPU faults into healthy nodes

    AITogether AI describes how its auto repair system handles GPU cluster faults, such as Xid errors, ECC errors and InfiniBand link flapping. Detection runs both inside the VM and on the host, and each fault maps to the lightest repair action, with customers choosing whether repairs wait for approval or run automatically. Repairs escalate from reboot to reprovision to migration when a fault recurs, and migrated machines are re-validated automatically before returning to the pool.

Heat Trend

Current comparable heat
11
Peak comparable heat
15Oct 27, 7:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.