Another great source, and a VERY interesting history of early magic shops is the chapter Laboratories of Legerdemain in Henry Ridgley Evans book History of Conjuring and Magic (1928). The chapter is about 12 pages long.
Evans points to Joseph Michael Hartz, along with his brother Augustus as being the first in New York to opening its shop (Hartz Magical Repository) a few years after the Civil War (1870 according to Whaley). Another brother, George Hartz, along with a Mr. John Levy opened a rival establishment, Hartz & Levy a year later, but it was very short lived.
According to Whaleys Whos Who in Magic, Nathaniel E. Chase opened the first magic shop in America in Boston in 1851, the second, also of Boston being Adams and Company in 1857. I wonder if Chase's was truly a storefront shop at that time?
Whaley references a great article in the June 1997 issue of The Linking Ring (by Bill Weldon) titled, The Hartz Brothers and Americas First Magic Shop. I just pulled the issue and read it, and it is filled with much interesting information.
Back to the original question, what was the first magic shop in America? Depends on what source you believe. Albo states in his Vol. XI Laboratories of Legerdemain that in an 1873 Chase catalog, it says, established 1851. My vote is for Chase based on that.
Cheers, Mark Damon
P.S. The Albo book (vol XI) is a must if you are interested in the history of magic shops. It covers America, Germany, England and France. They are also available as single soft bounds volumes on the used market, as they are now long out of print.