First visit products/display tab under woocommerce/settings menu and define custom product swatches height and width there.



Then select custom as display type while configuring into variation select tab.