The function is called fact(#1) without the key word function. Also the result needs to be feed back to TeX, e.g. via tex.write:
\documentclass{article}\directlua{dofile("luatest.lua")}\newcommand*{\myluafact}[1]{% \directlua{tex.write(fact(#1))}%}\begin{document} test \myluafact{5}\end{document}Image may be NSFW.
Clik here to view.
Improvement for \myluafact: The lua function fact expects a plain integer number, but TeX numbers can come in many different shapes (counters, registers, character constants, ...). If #1 is wrapped inside \numexpr, then even arithmetic calculations are possible:
\documentclass{article}\directlua{dofile("luatest.lua")}\newcommand*{\myluafact}[1]{% \directlua{tex.write(fact(\the\numexpr(#1)\relax))}%}\begin{document} test \myluafact{\value{page}+2*2}\end{document}