We study insertions of composite operators into Wilson loops in N=4 supersymmetric Yang-Mills theory in four dimensions. The loops follow a circular or straight path and the composite insertions transform in the adjoint representation of the gauge group. This provides a gauge invariant way to define the correlator of non-singlet operators. Since the basic loop preserves an SL(2,R) subgroup of the conformal group, we can assign a conformal dimension to those insertions and calculate the corrections to the classical dimension in perturbation theory. The calculation turns out to be very similar to that of single-trace local operators and may also be expressed in terms of a spin-chain. In this case the spin-chain is open and at one-loop order has Neumann boundary conditions on the type of scalar insertions that we consider. This system is integrable and we write the Bethe ansatz describing it. We compare the spectrum in the limit of large angular momentum both in the dilute gas approximation and the thermodynamic limit to the relevant string solution in the BMN limit and in the full AdS_5 x S^5 metric and find agreement.