.htaccess directives affect that directory, and all sorts of subdirectories thereof, which means you should disallow access inside your DocumentRoot,

http://sub.mydomain.com/.htaccess:

Order deny,allow
Deny from all

And override that in a specific subdirectories you want to allow use of,

http://sub.mydomain.com/test/.htaccess:

Order allow,deny
Allow from all

Try create .htaccess file in sub.mydomain.com for deny, as well as in sub.mydomain.com/test for allow.

Or redirect from http://sub.mydomain.com/ to deny subdir.

What about a .htaccess at the bottom directory, using the following lines?

RewriteEngine On
# check if request is for subdomain
RewriteCond %{HTTP_HOST} ^sub.mydomain.com$ [NC]
# check if  'test' isnt part of request
RewriteCond %{REQUEST_URI} !^/test/?(.*)$ [NC]
# if subdomain and no 'test' part, redirect to main domain...
RewriteRule ^(.*)$ http://www.mydomain.com/$1 [R,L]

Therefore if the '/test/' section exists, no redirect happens...