I agree with a lot of what you're saying. I think it's more that those moments of belief come and go. And that his sojourn with Angel kept chipping away at his confidence without any correlating interaction with Buffy. While he may have believed in Chosen, I think that belief fades away every moment he's away from her and with Angel telling him he doesn't matter.