According to On Location Tours, Buddy’s visit to his father’s office in Elf was set at the Empire State Building. The tour operator is the source for this, since Wikipedia does not name New York sites for the film.
It is a public building, so check its own site for visiting rules and ticket costs.
See our Christmas movie filming locations guide for other New York sites.