Learning a second language (L2) at a young age is a driving factor of functional neuroplasticity in the auditory brainstem. To date, it remains unclear whether these effects remain stable until adulthood and to what degree the amount of exposure to the L2 in early childhood might affect their outcome. We compared three groups of adult English-French bilinguals in their ability to categorize English vowels in relation to their frequency following responses (FFR) evoked by the same vowels. At the time of testing, cognitive abilities as well as fluency in both languages were matched between the (1) simultaneous bilinguals (SIM, N = 18); (2) sequential bilinguals with L1-English (N = 14); and (3) sequential bilinguals with L1-French (N = 11). Our results show that the L1-English group show sharper category boundaries in identification of the vowels compared to the L1-French group. Furthermore, the same pattern was reflected in the FFRs (i.e., larger FFR responses in L1-English > SIM > L1-French), while again only the difference between the L1-English and the L1-French group was statistically significant; nonetheless, there was a trend towards larger FFR in SIM compared to L1-French. Our data extends previous literature showing that exposure to a language during the first years of life induces functional neuroplasticity in the auditory brainstem that remains stable until at least young adulthood. Furthermore, the findings suggest that amount of exposure (i.e., 100% vs. 50%) to that language does not differentially shape the robustness of the perceptual abilities or the auditory brainstem encoding of phonetic categories of the language. Statement of significance: Previous studies have indicated that early age of L2 acquisition induces functional neuroplasticity in the auditory brainstem during processing of the L2. This study compared three groups of adult bilinguals who differed in their age of L2 acquisition as well as the amount of exposure to the L2 during early childhood. We demonstrate for the first time that the neuroplastic effect in the brainstem remains stable until young adulthood and that the amount of L2 exposure does not influence behavioral or brainstem plasticity. Our study provides novel insights into low-level auditory plasticity as a function of varying bilingual experience.