Should colors and sizes be categories?

Usually no. Colors, sizes, dimensions, materials, and similar details often work better as attributes or custom fields that can power filters.