Well we are talking about casual wear. I surely hope you would not expect a man to wear slacks/suit/dress shoes on daily errands or going out for lunch? Those would be strong standards :p
Pointed toes heels are only reserved for formal occasions IMO. Red carpet/Little black dress type outfit. I'm not even a fan of them with skinny jeans. Stick to the basics with heels :)