I can relate to this a bit. When I finished grad studies in SG, I moved outside for few years to work. When I moved back, I took a 35% pay cut for a startup mainly due to family reasons.At that time, CS was not too hot here too unlike now. However, that was my best job cuz I worked with people in Boston and worked on ML/Big Data. Its so useful to learn from right people. But it also sucked to earn less. After 2.5 yrs, suddenly I was in demand and got offers often, even FAANG types. My salary almost became 4 times in just 4-5 years. But even though the initial moves paid more than the startup, they were run by typical sales/people managers who cannot teach anyone jack shit.
For me what works, is knowledge n constant improvement. Small hiccups are not gonna ruin your career.
I hope your job adds value to skills - whether hard skills or soft. If that happens, you don't have to worry.
And, I would not compromise on my dreams just bcuz I am scared of blacklist or termed a job-hopper.
Follow you heart and Don't take Advice