Either get more memory or try this env property:

#Set the number of files the database can have open at one time db.nosql.maxOpenFiles=500

It will close/unmap files at the expense of cpu use. If this problem persists then try another stream type:

#Query Performance Tuning, File Access Type:Available[INPUT_STREAM,FILE_CHANNEL,RANDOM_ACCESS_FILE,MAPPED_BYTE_BUFFER] db.nosql.shardStreamType=MAPPED_BYTE_BUFFER