More than half of people in Wales have seen a decline in their physical health over the last three years.