Determining individual-level interactions that govern highly coordinated motion in animal groups or cellular aggregates has been a long-standing challenge, central to understanding the mechanisms and evolution of collective behavior. Numerous models have been proposed, many of which display realistic-looking dynamics, but nonetheless rely on untested assumptions about how individuals integrate information to guide movement. Here we infer behavioral rules directly from experimental data. We begin by analyzing trajectories of golden shiners (Notemigonus crysoleucas) swimming in two-fish and three-fish shoals to map the mean effective forces as a function of fish positions and velocities. Speeding and turning responses are dynamically modulated and clearly delineated. Speed regulation is a dominant component of how fish interact, and changes in speed are transmitted to those both behind and ahead. Alignment emerges from attraction and repulsion, and fish tend to copy directional changes made by those ahead. We find no evidence for explicit matching of body orientation. By comparing data from two-fish and three-fish shoals, we challenge the standard assumption, ubiquitous in physics-inspired models of collective behavior, that individual motion results from averaging responses to each neighbor considered separately; three-body interactions make a substantial contribution to fish dynamics. However, pairwise interactions qualitatively capture the correct spatial interaction structure in small groups, and this structure persists in larger groups of 10 and 30 fish. The interactions revealed here may help account for the rapid changes in speed and direction that enable real animal groups to stay cohesive and amplify important social information.