Without a doubt your team lacks a true PG, as you mentioned. Honestly, I like Conley at the PG position, because of his ability to create turnovers from the opposing team's PG. Their PG had turned the ball 21 times, in 4 games including the two exhibition games. All 3 of the guys seem pretty close/comparable in terms of BH/Pass. However the one thing that might concern me with conley is the lack of speed. In Conley's first two regular season games, he has a 11/3 assist to turnover ratio. Unless Conley can create something off of his athleticism, at least right now you don't have much of a scorer, which makes Conley perfect for the position.
I like Perkins at SG because of his 88-93 speed/perimeter rating, not sure how much is lack of ballhandling will effect that. Ray is the perfect scoring SF, his PE+LP skills are pretty astounding for D-3.
And 41 isn't a terrible defensive rating for D-3, just because it isn't 60 or 70 like most players, doesn't mean it's terrible either. If he has some offensive talent, the 41 DE can be forgiven.