Going to lunch with a colleague?

I am 23F. I just got an entry job at a loan agency. One of my coworkers, well, he's one of the seniors at the company who is around his late 50s and early 60s. He's married with children. Super friendly guy. We talk all the time and he's just a super talkative guy who I absolutely enjoy chit chatting with. It's weird though how he only talks long convos with me and none of my other entry level coworkers. Recently when I told him that I'm actually super antisocial, he was quite surprised, and told me that he would've never guessed because I'm always so bright and outgoing.

One thing is that he told me that we should go to sushi for lunch sometime and I always said yes to it but we never planned it. He keeps bringing up in a super friendly way too but he never forces it. My question is, is it okay for me to go to lunch with my coworker? Either during office hours or outside of work, is this okay? Why is he wanting to take me to lunch? Not that I'm uncomfortable, but I have never been in a situation like this before.