WordPress Taxonomy Template Pagination Issue - php

Question

After dedicating hours on this issue I finally decided to ask the community about the problem I am facing. This issue itself is somewhat interesting and strange.
I created a custom taxonomy template and while I am using the default wordpress loop I am facing issues in Pagination. The posts in every page are coming randomly.
So lets say if I am on page 1 of the taxonomy pagination it shows me 2 posts and if I go to page 2 it shows me 4 posts then 2 and then 3 posts per page ..
I have set the posts_per_page as 6 using pre_get_posts action on my functions.php file ..
I am giving the brief codes below ..
TEMPLATE FILE
<?php get_header(); ?>
<?php
global $wp_query;
$big = 999999999; // need an unlikely integer
$paged = ( get_query_var('paged') ) ? get_query_var('paged') : 1;
?>
<div <?php post_class('main-section page-inner-wrapper'); ?>>
<?php get_template_part('talent_filter', 'talent_filter.php'); ?>
<?php if ( have_posts() ) : ?>
<?php while ( have_posts() ) : the_post(); ?>
<!-- POST HTML -->
<?php endwhile; ?>
<!-- PAGINATION BEGIN -->
<nav class="navigation pagination" role="navigation">
<?php
echo paginate_links( array(
'base' => str_replace( $big, '%#%', esc_url( get_pagenum_link( $big ) ) ),
'format' => '?paged=%#%',
'current' => $paged,
'prev_text' => __( '<i class="fa fa-angle-left"></i>', 'bridge' ),
'next_text' => __( '<i class="fa fa-angle-right"></i>', 'bridge' ),
'before_page_number' => '<span class="meta-nav">' . __( '', 'bridge' ) . ' </span>',
'total' => $wp_query->max_num_pages
) );
?>
</nav>
<!-- PAGINATION END -->
<?php endif; ?>
</div>
<?php get_footer(); ?>
FUNCTIONS IN function.php
function my_post_queries( $query ) {
// do not alter the query on wp-admin pages and only alter it if it's the main query
if (!is_admin() && $query->is_main_query()){
// alter the query for the home and category pages
if( is_tax( 'talent_groups' ) ) {
$query->set('posts_per_page', 6);
$query->set('posts_per_archive_page', 6);
$query->set('orderby', 'date');
}
}
}
add_action( 'pre_get_posts', 'my_post_queries');
// Solves Pagination Pages 404's
add_filter( 'option_posts_per_page', 'tdd_tax_filter_posts_per_page' );
function tdd_tax_filter_posts_per_page( $value ) {
return (is_tax('talent_groups')) ? 1 : $value;
}