In Views, you can order posts by the distance from a given address or the current user location, not by the address set as centre in the front end when querying by distance.
This code snippet will allow passing the queried centre for the distance filter in the front end to the orderby settings of the View, so updating the resulting results order with the new centre.

add_filter( 'wpv_view_settings', 'prefix_modify_rendered_view', 30, 2 );
function prefix_modify_rendered_view( $view_settings, $view_id ) {
    if ( $view_id == 51 ) { // if displaying a View with ID equal to 23
        if ($_GET['toolset_maps_distance_center'] != '') {
            $query_center = $_GET['toolset_maps_distance_center'];
            $view_settings['distance_order'] = array(
                "source" => "fixed",
                "center" => $query_center
            );
        }
    return $view_settings;
}
}

Note, there might be better ways to the URL param.