Friends with benefits or something more?

I have had this best friend since high school and we are now in college. It's my second year but he is already working this year.

He had a girl friend who was my friend too. I never went along much with her but as no one wanted to be with her, i always hung out with her. They both were together in the last year of our high school, where the girl just wanted to play around with him. But they both ended up having something deep. I used to be a bit jealous but did not let it affect me too much.

When in college, the girl started having secret affairs with some other guy. I knew everything but i never told her boyfriend who was my best friend. Things started getting weird. His boyfriend knew that she was that way, they had break ups few times but the girl always could melt him down and get back to him again.

After this too much getting on and breaking up, me and my best friend we started hanging out and drinking. We ended up having a physical relationship, maybe over limit. He still loved his girl a lot. I couldn't see it much of them being together so i just said yes to a guy who proposed me last year and i stopped talking to my best friend because i thought we may end up nothing and i may be a pain because of their relationship.

I dont know what the real case was but many of my other friends said that he can't take the reality that i am with some other guy and i am giving all my time to my boyfriend and not him.

Days passed, he started giving me calls, and i could not move back. And once again we were in contact. We started meeting secretly, everything started again and went too far.

I broke up with my boyfriend, he broke up with his girlfriend. Its one year already, we both are single. However, we still have everything like a girl friend and boyfriend have, i feel we have love in between us, the way we care for each other, the way we behave with each other, the way we give time to each other. It's like already so perfect.

But just a few days ago, we ended up in this conversation. Where i asked him will i never be in relationship with you? and he just replied saying i never thought of you more than a friend.

Before when he broke up, he was really hurt, i just couldn't control him and his tears and his pain. But i gave all my best to make him happy, he was but somewhere he still missed his girlfriend and that's the reason i never asked him for any relationship. But i really want us to be together.

Since he told me i never thought of you more than friends, i am not talking to him properly. I mean, i am but it's just i wanted something else. If we can come so far, with everything in between us, then why can't we be together?

I really need help, i don't know if what i am doing is right or wrong.