A bid to put up a new building on agricultural land for non-agricultural use by the owner of Gloucester Asbestos Ltd has been rejected by planners.