Why did Malcolm X change his second name, but not his first?

He said his english family name Little was imposed upon his ancestors by slavemasters and was not natural. But Malcolm is english too, so why didn't he change that one? I know he also had an arabic name - El-Hajj Malik El-Shabazz - but he didn't use that in the media.