Solution #10

From xy + x + y = 71, we have xy + x + y + 1 = 72, or (x+1)(y+1) = 72.

From x²y+xy² = 880 we have xy(x+y) = 880.

880 = 2⁴*5*11, so each of x, y, and x+y must have its factors among these.

If (x+1)(y+1) = 72, then the pair of x+1 and y+1 must be (in some order), one of (1,72), (2,36), (3,24), (4,18), (6,12), or (8,9).

These pairs then give possible values for x and y of (0,71), (1,35), (2,23), (3,17), (5,11), and (7,8). (0,71) can be eliminated because x and y are both positive. (1,35), (2,23), (3,17), and (7,8) all include prime factors for x or y not in our permitted list. Thus we must have x and y be 5 and 11. x² + y² is then 121 + 25 = 146.

(Note that our answer does not tell us which of x and y is 5, and which is 11. This is to be expected, given the symmetry of the starting information in x and y.)