Why did Malcolm X change his family name, but not his first name?

He said his english family name "Little" was imposed upon his ancestors by slavemasters and was not natural. But Malcolm is english too, so why didn't he change that one? I know he also had an arabic name - El-Hajj Malik El-Shabazz - but he didn't use that in the media.