A 3.0-kg mass is positioned at (0, 8.0), and a 1.0-kg mass is positioned at (12, 0). What are the coordinates of a 4.0-kg mass which will result in the center of mass of the system of three masses being located at the origin, (0, 0)?

Newton's second law in terms of momentum:  The rate of change of momentum of an object is equal to the net force applied to it.
The law of conservation of momentum: The total momentum of an isolated system of objects remains constant.

Newton's second law for a system of particles: The sum of all forces acting on the system is equal to the total mass of the system times the acceleration of its center of mass.