Do we pass the user-supplied fields object to calendar.*FromFields, or do we make a new object? If we pass the user-supplied one directly, we shouldn't bother with removing the calendar. If we re-create a new object, then I think it makes sense to remove the calendar field.
From a quick grep, it seems like only the from methods pass through a user-supplied object (and I'd probably prefer to change those as well for consistency).