Hey Rdrose - looking good!
About your temps in your room, they need to come down and stay between 80* to 85*. You need more air exchanges in your space to remove the heat.
First off, your ballast needs to be out of your grow area as its uneccesary heat. Second, shorten all "flex" ducting like you did. Flex duct is very restrictive to airflow when it's spaghetti'ed all over the place.
You are pulling air out of your room, and not "blowing" it in, correct? You always want to pull air out of your space. Where is the air from the exhaust going? Into the attic?
When choosing exhaust fans, I prefer the "centrifigal" type instead of the "axial" or propeller types. They are able to move more air with restrictive duct runs. Some people run their exhaust fan all the time, but I have mine operated by a thermostat for efficiency. Here is a link to the perfect plug in type thermostat for your exhaust fan. It is a duct mounted one, but can be mounted on any surface to maintain a room temp.
I use 2 thermostats in my grow area. One for the exhaust fan, and the other for a recirculation fan inside the box. This eliminates hot spots inside. Temps are real even everywhere.
http://www.iaqsource.com/product.php?product=111134
Here is the manuf website for the thermostats. Also other stuff you may need is in there too.
http://www.suncourt.com/Hydroponics.html
The only other thing I can think of is you need a hole large enough in the closet to let cool air in, near the bottom of the area. You can't pull air out of a space if you don't have a place for it to come in. A 6" diam hole would be large enough. You may need to make a "light baffle" for stealth outside the grow if you need.