Multiple potential claims:
1. DMCA § 1201 (anti-circumvention): If they bypassed any technical protection measures to access your code or file format, this provides a federal claim regardless of fair use.
2. Trade secret: If your file format and internal logic aren't publicly documented, they may qualify as trade secrets. Reverse engineering CAN be a legitimate means of discovering trade secrets in some jurisdictions, but EULA restrictions may change the analysis.
3. Copyright: If they copied substantial code rather than just observing behavior and reimplementing, that's infringement. But post-Google v. Oracle, the fair use defense for interoperability-focused copying is stronger.
4. EULA breach: If they accepted your EULA (by installing your software) and then violated the reverse-engineering prohibition, that's breach of contract. Enforceability of anti-RE clauses varies by jurisdiction — some courts have held them unenforceable as against public policy (interoperability).
Strongest approach: combine trade secret claims (if the file format was confidential) with DMCA anti-circumvention. These don't have the "fair use" defense that weakens copyright claims.